powerpc/64s: Leave interrupts hard enabled in context switch for radix

Commit 4387e9ff25 ("[POWERPC] Fix PMU + soft interrupt disable bug")
hard disabled interrupts over the low level context switch, because
the SLB management can't cope with a PMU interrupt accesing the stack
in that window.

Radix based kernel mapping does not use the SLB so it does not require
interrupts hard disabled here.

This is worth 1-2% in context switch performance on POWER9.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>

diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index 6f70ea821a07d86955a80be14dcd44d42f1da005..91f9fdc2d027df702c5d2768f215fef2f01ba90c 100644 (file)
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -607,6 +607,14 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT)
           top of the kernel stack. */
        addi    r7,r7,THREAD_SIZE-SWITCH_FRAME_SIZE
 
+       /*
+        * PMU interrupts in radix may come in here. They will use r1, not
+        * PACAKSAVE, so this stack switch will not cause a problem. They
+        * will store to the process stack, which may then be migrated to
+        * another CPU. However the rq lock release on this CPU paired with
+        * the rq lock acquire on the new CPU before the stack becomes
+        * active on the new CPU, will order those stores.
+        */
        mr      r1,r8           /* start using new stack pointer */
        std     r7,PACAKSAVE(r13)
 

diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 5cbb8b1faf7e5073262381e76ddb975c1f845ed5..45faa9a32a0190325908e5986daed62972bf4296 100644 (file)
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1199,12 +1199,14 @@ struct task_struct *__switch_to(struct task_struct *prev,
 
        __switch_to_tm(prev, new);
 
-       /*
-        * We can't take a PMU exception inside _switch() since there is a
-        * window where the kernel stack SLB and the kernel stack are out
-        * of sync. Hard disable here.
-        */
-       hard_irq_disable();
+       if (!radix_enabled()) {
+               /*
+                * We can't take a PMU exception inside _switch() since there
+                * is a window where the kernel stack SLB and the kernel stack
+                * are out of sync. Hard disable here.
+                */
+               hard_irq_disable();
+       }
 
        /*
         * Call restore_sprs() before calling _switch(). If we move it after