perf/x86/intel/pebs: Robustify PEBS buffer drain

Vince Weaver and Stephane Eranian reported warnings in the PEBS
code when running the perf fuzzer. Stephane wrote:

  > I can reproduce the problem on my HSW running the fuzzer.
  >
  > I can see why this could be happening if you are mixing PEBS and non PEBS events
  > in the bottom 4 counters. I suspect:
  >         for (bit = 0; bit < x86_pmu.max_pebs_events; bit++) {
  >                 if ((counts[bit] == 0) && (error[bit] == 0))
  >                         continue;
  >
  > This test is not correct when you have non-PEBS events mixed with
  > PEBS events and they overflow at the same time. They will have
  > counts[i] != 0 but error[i] == 0, and thus you fall thru the loop
  > and hit the assert. Or it is something along those lines.

The only way I can make this work is if ->status only has !PEBS events
set, because if it has both set we'll take that slow path which masks
out the !PEBS bits.

After masking there are 3 options:

 - there is one bit set, and its @bit, we increment counts[bit].

 - there are multiple bits set, we increment error[] for each set bit,
   we do not increment counts[].

 - there are no bits set, we do nothing.

The intent was to never increment counts[] for !PEBS events.

Now if we start out with only a single !PEBS event set, we'll pass the
test and increment counts[] for a !PEBS and hit the warn.

Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Reported-by: Stephane Eranian <eranian@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: kan.liang@intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>

diff --git a/arch/x86/kernel/cpu/perf_event_intel_ds.c b/arch/x86/kernel/cpu/perf_event_intel_ds.c
index 8e564f817a7f155ab635ff460b84afbbe67e24fe..84f236ab96b03459907662de8842958a3645d011 100644 (file)
--- a/arch/x86/kernel/cpu/perf_event_intel_ds.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_ds.c
@@ -1188,6 +1188,7 @@ static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs)
 
        for (at = base; at < top; at += x86_pmu.pebs_record_size) {
                struct pebs_record_nhm *p = at;
+               u64 pebs_status;
 
                /* PEBS v3 has accurate status bits */
                if (x86_pmu.intel_cap.pebs_format >= 3) {
@@ -1198,12 +1199,17 @@ static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs)
                        continue;
                }
 
-               bit = find_first_bit((unsigned long *)&p->status,
+               pebs_status = p->status & cpuc->pebs_enabled;
+               pebs_status &= (1ULL << x86_pmu.max_pebs_events) - 1;
+
+               bit = find_first_bit((unsigned long *)&pebs_status,
                                        x86_pmu.max_pebs_events);
-               if (bit >= x86_pmu.max_pebs_events)
-                       continue;
-               if (!test_bit(bit, cpuc->active_mask))
+               if (WARN(bit >= x86_pmu.max_pebs_events,
+                        "PEBS record without PEBS event! status=%Lx pebs_enabled=%Lx active_mask=%Lx",
+                        (unsigned long long)p->status, (unsigned long long)cpuc->pebs_enabled,
+                        *(unsigned long long *)cpuc->active_mask))
                        continue;
+
                /*
                 * The PEBS hardware does not deal well with the situation
                 * when events happen near to each other and multiple bits
@@ -1218,27 +1224,21 @@ static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs)
                 * one, and it's not possible to reconstruct all events
                 * that caused the PEBS record. It's called collision.
                 * If collision happened, the record will be dropped.
-                *
                 */
-               if (p->status != (1 << bit)) {
-                       u64 pebs_status;
-
-                       /* slow path */
-                       pebs_status = p->status & cpuc->pebs_enabled;
-                       pebs_status &= (1ULL << MAX_PEBS_EVENTS) - 1;
-                       if (pebs_status != (1 << bit)) {
-                               for_each_set_bit(i, (unsigned long *)&pebs_status,
-                                                MAX_PEBS_EVENTS)
-                                       error[i]++;
-                               continue;
-                       }
+               if (p->status != (1ULL << bit)) {
+                       for_each_set_bit(i, (unsigned long *)&pebs_status,
+                                        x86_pmu.max_pebs_events)
+                               error[i]++;
+                       continue;
                }
+
                counts[bit]++;
        }
 
        for (bit = 0; bit < x86_pmu.max_pebs_events; bit++) {
                if ((counts[bit] == 0) && (error[bit] == 0))
                        continue;
+
                event = cpuc->events[bit];
                WARN_ON_ONCE(!event);
                WARN_ON_ONCE(!event->attr.precise_ip);