performancecounter - Use linux perf utility to report counters every second like vmstat -

there perf command-linux utility in linux access hardware performance-monitoring counters, works using perf_events kernel subsystems.

perf has 2 modes: perf record/perf top record sampling profile (the sample example every 100000th cpu clock cycle or executed command), , perf stat mode report total count of cycles/executed commands application (or whole system).

is there mode of perf print system-wide or per-cpu summary on total count every second (every 3, 5, 10 seconds), printed in vmstat , systat-family tools (iostat, mpstat, sar -n dev... listed in http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html)? example, cycles , instructions counters mean ipc every second of system (or of every cpu).

is there non-perf tool (in https://perf.wiki.kernel.org/index.php/tutorial or http://www.brendangregg.com/perf.html) can such statistics perf_events kernel subsystem? system-wide per-process ipc calculation resolution of seconds?

there perf stat option "interval-print" of -i n n millisecond interval interval counter printing every n milliseconds (n>=10): http://man7.org/linux/man-pages/man1/perf-stat.1.html

  -i msecs, --interval-print msecs        print count deltas every n milliseconds (minimum: 10ms)        overhead percentage high in cases, instance        small, sub 100ms intervals. use caution. example: perf        stat -i 1000 -e cycles -a sleep 5    best results idea use interval    mode -i 1000, bottleneck of workloads can change often. 

there importing results in machine-readable form, , -i first field datetime:

with -x, perf stat able output not-quite-csv format output ... optional usec time stamp in fractions of second (with -i xxx)

vmstat, systat-family tools iostat, mpstat, etc periodic printing -i 1000 of perf stat (every second), example system-wide (add -a separate cpu counters):

  perf stat -a -i 1000 

the option implemented in builtin-stat.c http://lxr.free-electrons.com/source/tools/perf/builtin-stat.c?v=4.8 __run_perf_stat function

531 static int __run_perf_stat(int argc, const char **argv) 532 { 533         int interval = stat_config.interval; 

for perf stat -i 1000 program argument (forks=1), example perf stat -i 1000 sleep 10 there interval loop (ts millisecond interval converted struct timespec):

639                 enable_counters(); 641                 if (interval) { 642                         while (!waitpid(child_pid, &status, wnohang)) { 643                                 nanosleep(&ts, null); 644                                 process_interval(); 645                         } 646                 } 666         disable_counters(); 

for variant of system-wide hardware performance monitor counting , forks=0 there other interval loop

658                 enable_counters(); 659                 while (!done) { 660                         nanosleep(&ts, null); 661                         if (interval) 662                                 process_interval(); 663                 } 666         disable_counters(); 

process_interval() http://lxr.free-electrons.com/source/tools/perf/builtin-stat.c?v=4.8#l347 same file uses read_counters(); loops on event list , invokes read_counter() loops over known threads , cpus , starts actual reading function:

306         (thread = 0; thread < nthreads; thread++) { 307                 (cpu = 0; cpu < ncpus; cpu++) { ... 310                         count = perf_counts(counter->counts, cpu, thread); 311                         if (perf_evsel__read(counter, cpu, thread, count)) 312                                 return -1; 

perf_evsel__read real counter read while program still running:

1207 int perf_evsel__read(struct perf_evsel *evsel, int cpu, int thread, 1208                      struct perf_counts_values *count) 1209 { 1210         memset(count, 0, sizeof(*count)); 1211  1212         if (fd(evsel, cpu, thread) < 0) 1213                 return -einval; 1214  1215         if (readn(fd(evsel, cpu, thread), count, sizeof(*count)) < 0) 1216                 return -errno; 1217  1218         return 0; 1219 }