|
@@ -524,7 +524,7 @@ The following commands are supported:
|
|
|
|
|
|
hist:keys=<field1[,field2,...]>[:values=<field1[,field2,...]>]
|
|
|
[:sort=<field1[,field2,...]>][:size=#entries][:pause][:continue]
|
|
|
- [:clear] [if <filter>]
|
|
|
+ [:clear][:name=histname1] [if <filter>]
|
|
|
|
|
|
When a matching event is hit, an entry is added to a hash table
|
|
|
using the key(s) and value(s) named. Keys and values correspond to
|
|
@@ -546,18 +546,28 @@ The following commands are supported:
|
|
|
specified by the 'sort' keyword. If more than one field is
|
|
|
specified, the result will be a 'sort within a sort': the first key
|
|
|
is taken to be the primary sort key and the second the secondary
|
|
|
- key.
|
|
|
+ key. If a hist trigger is given a name using the 'name' parameter,
|
|
|
+ its histogram data will be shared with other triggers of the same
|
|
|
+ name, and trigger hits will update this common data. Only triggers
|
|
|
+ with 'compatible' fields can be combined in this way; triggers are
|
|
|
+ 'compatible' if the fields named in the trigger share the same
|
|
|
+ number and type of fields and those fields also have the same names.
|
|
|
+ Note that any two events always share the compatible 'hitcount' and
|
|
|
+ 'stacktrace' fields and can therefore be combined using those
|
|
|
+ fields, however pointless that may be.
|
|
|
|
|
|
'hist' triggers add a 'hist' file to each event's subdirectory.
|
|
|
Reading the 'hist' file for the event will dump the hash table in
|
|
|
its entirety to stdout. If there are multiple hist triggers
|
|
|
attached to an event, there will be a table for each trigger in the
|
|
|
- output. Each printed hash table entry is a simple list of the keys
|
|
|
- and values comprising the entry; keys are printed first and are
|
|
|
- delineated by curly braces, and are followed by the set of value
|
|
|
- fields for the entry. By default, numeric fields are displayed as
|
|
|
- base-10 integers. This can be modified by appending any of the
|
|
|
- following modifiers to the field name:
|
|
|
+ output. The table displayed for a named trigger will be the same as
|
|
|
+ any other instance having the same name. Each printed hash table
|
|
|
+ entry is a simple list of the keys and values comprising the entry;
|
|
|
+ keys are printed first and are delineated by curly braces, and are
|
|
|
+ followed by the set of value fields for the entry. By default,
|
|
|
+ numeric fields are displayed as base-10 integers. This can be
|
|
|
+ modified by appending any of the following modifiers to the field
|
|
|
+ name:
|
|
|
|
|
|
.hex display a number as a hex value
|
|
|
.sym display an address as a symbol
|
|
@@ -1809,3 +1819,251 @@ The following commands are supported:
|
|
|
Hits: 0
|
|
|
Entries: 0
|
|
|
Dropped: 0
|
|
|
+
|
|
|
+ Named triggers can be used to have triggers share a common set of
|
|
|
+ histogram data. This capability is mostly useful for combining the
|
|
|
+ output of events generated by tracepoints contained inside inline
|
|
|
+ functions, but names can be used in a hist trigger on any event.
|
|
|
+ For example, these two triggers when hit will update the same 'len'
|
|
|
+ field in the shared 'foo' histogram data:
|
|
|
+
|
|
|
+ # echo 'hist:name=foo:keys=skbaddr.hex:vals=len' > \
|
|
|
+ /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger
|
|
|
+ # echo 'hist:name=foo:keys=skbaddr.hex:vals=len' > \
|
|
|
+ /sys/kernel/debug/tracing/events/net/netif_rx/trigger
|
|
|
+
|
|
|
+ You can see that they're updating common histogram data by reading
|
|
|
+ each event's hist files at the same time:
|
|
|
+
|
|
|
+ # cat /sys/kernel/debug/tracing/events/net/netif_receive_skb/hist;
|
|
|
+ cat /sys/kernel/debug/tracing/events/net/netif_rx/hist
|
|
|
+
|
|
|
+ # event histogram
|
|
|
+ #
|
|
|
+ # trigger info: hist:name=foo:keys=skbaddr.hex:vals=hitcount,len:sort=hitcount:size=2048 [active]
|
|
|
+ #
|
|
|
+
|
|
|
+ { skbaddr: ffff88000ad53500 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800af5a1500 } hitcount: 1 len: 76
|
|
|
+ { skbaddr: ffff8800d62a1900 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bccb00 } hitcount: 1 len: 468
|
|
|
+ { skbaddr: ffff8800d3c69900 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff88009ff09100 } hitcount: 1 len: 52
|
|
|
+ { skbaddr: ffff88010f13ab00 } hitcount: 1 len: 168
|
|
|
+ { skbaddr: ffff88006a54f400 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bcc500 } hitcount: 1 len: 260
|
|
|
+ { skbaddr: ffff880064505000 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800baf24e00 } hitcount: 1 len: 32
|
|
|
+ { skbaddr: ffff88009fe0ad00 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d3edff00 } hitcount: 1 len: 44
|
|
|
+ { skbaddr: ffff88009fe0b400 } hitcount: 1 len: 168
|
|
|
+ { skbaddr: ffff8800a1c55a00 } hitcount: 1 len: 40
|
|
|
+ { skbaddr: ffff8800d2bcd100 } hitcount: 1 len: 40
|
|
|
+ { skbaddr: ffff880064505f00 } hitcount: 1 len: 174
|
|
|
+ { skbaddr: ffff8800a8bff200 } hitcount: 1 len: 160
|
|
|
+ { skbaddr: ffff880044e3cc00 } hitcount: 1 len: 76
|
|
|
+ { skbaddr: ffff8800a8bfe700 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bcdc00 } hitcount: 1 len: 32
|
|
|
+ { skbaddr: ffff8800a1f64800 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bcde00 } hitcount: 1 len: 988
|
|
|
+ { skbaddr: ffff88006a5dea00 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff88002e37a200 } hitcount: 1 len: 44
|
|
|
+ { skbaddr: ffff8800a1f32c00 } hitcount: 2 len: 676
|
|
|
+ { skbaddr: ffff88000ad52600 } hitcount: 2 len: 107
|
|
|
+ { skbaddr: ffff8800a1f91e00 } hitcount: 2 len: 92
|
|
|
+ { skbaddr: ffff8800af5a0200 } hitcount: 2 len: 142
|
|
|
+ { skbaddr: ffff8800d2bcc600 } hitcount: 2 len: 220
|
|
|
+ { skbaddr: ffff8800ba36f500 } hitcount: 2 len: 92
|
|
|
+ { skbaddr: ffff8800d021f800 } hitcount: 2 len: 92
|
|
|
+ { skbaddr: ffff8800a1f33600 } hitcount: 2 len: 675
|
|
|
+ { skbaddr: ffff8800a8bfff00 } hitcount: 3 len: 138
|
|
|
+ { skbaddr: ffff8800d62a1300 } hitcount: 3 len: 138
|
|
|
+ { skbaddr: ffff88002e37a100 } hitcount: 4 len: 184
|
|
|
+ { skbaddr: ffff880064504400 } hitcount: 4 len: 184
|
|
|
+ { skbaddr: ffff8800a8bfec00 } hitcount: 4 len: 184
|
|
|
+ { skbaddr: ffff88000ad53700 } hitcount: 5 len: 230
|
|
|
+ { skbaddr: ffff8800d2bcdb00 } hitcount: 5 len: 196
|
|
|
+ { skbaddr: ffff8800a1f90000 } hitcount: 6 len: 276
|
|
|
+ { skbaddr: ffff88006a54f900 } hitcount: 6 len: 276
|
|
|
+
|
|
|
+ Totals:
|
|
|
+ Hits: 81
|
|
|
+ Entries: 42
|
|
|
+ Dropped: 0
|
|
|
+ # event histogram
|
|
|
+ #
|
|
|
+ # trigger info: hist:name=foo:keys=skbaddr.hex:vals=hitcount,len:sort=hitcount:size=2048 [active]
|
|
|
+ #
|
|
|
+
|
|
|
+ { skbaddr: ffff88000ad53500 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800af5a1500 } hitcount: 1 len: 76
|
|
|
+ { skbaddr: ffff8800d62a1900 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bccb00 } hitcount: 1 len: 468
|
|
|
+ { skbaddr: ffff8800d3c69900 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff88009ff09100 } hitcount: 1 len: 52
|
|
|
+ { skbaddr: ffff88010f13ab00 } hitcount: 1 len: 168
|
|
|
+ { skbaddr: ffff88006a54f400 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bcc500 } hitcount: 1 len: 260
|
|
|
+ { skbaddr: ffff880064505000 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800baf24e00 } hitcount: 1 len: 32
|
|
|
+ { skbaddr: ffff88009fe0ad00 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d3edff00 } hitcount: 1 len: 44
|
|
|
+ { skbaddr: ffff88009fe0b400 } hitcount: 1 len: 168
|
|
|
+ { skbaddr: ffff8800a1c55a00 } hitcount: 1 len: 40
|
|
|
+ { skbaddr: ffff8800d2bcd100 } hitcount: 1 len: 40
|
|
|
+ { skbaddr: ffff880064505f00 } hitcount: 1 len: 174
|
|
|
+ { skbaddr: ffff8800a8bff200 } hitcount: 1 len: 160
|
|
|
+ { skbaddr: ffff880044e3cc00 } hitcount: 1 len: 76
|
|
|
+ { skbaddr: ffff8800a8bfe700 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bcdc00 } hitcount: 1 len: 32
|
|
|
+ { skbaddr: ffff8800a1f64800 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff8800d2bcde00 } hitcount: 1 len: 988
|
|
|
+ { skbaddr: ffff88006a5dea00 } hitcount: 1 len: 46
|
|
|
+ { skbaddr: ffff88002e37a200 } hitcount: 1 len: 44
|
|
|
+ { skbaddr: ffff8800a1f32c00 } hitcount: 2 len: 676
|
|
|
+ { skbaddr: ffff88000ad52600 } hitcount: 2 len: 107
|
|
|
+ { skbaddr: ffff8800a1f91e00 } hitcount: 2 len: 92
|
|
|
+ { skbaddr: ffff8800af5a0200 } hitcount: 2 len: 142
|
|
|
+ { skbaddr: ffff8800d2bcc600 } hitcount: 2 len: 220
|
|
|
+ { skbaddr: ffff8800ba36f500 } hitcount: 2 len: 92
|
|
|
+ { skbaddr: ffff8800d021f800 } hitcount: 2 len: 92
|
|
|
+ { skbaddr: ffff8800a1f33600 } hitcount: 2 len: 675
|
|
|
+ { skbaddr: ffff8800a8bfff00 } hitcount: 3 len: 138
|
|
|
+ { skbaddr: ffff8800d62a1300 } hitcount: 3 len: 138
|
|
|
+ { skbaddr: ffff88002e37a100 } hitcount: 4 len: 184
|
|
|
+ { skbaddr: ffff880064504400 } hitcount: 4 len: 184
|
|
|
+ { skbaddr: ffff8800a8bfec00 } hitcount: 4 len: 184
|
|
|
+ { skbaddr: ffff88000ad53700 } hitcount: 5 len: 230
|
|
|
+ { skbaddr: ffff8800d2bcdb00 } hitcount: 5 len: 196
|
|
|
+ { skbaddr: ffff8800a1f90000 } hitcount: 6 len: 276
|
|
|
+ { skbaddr: ffff88006a54f900 } hitcount: 6 len: 276
|
|
|
+
|
|
|
+ Totals:
|
|
|
+ Hits: 81
|
|
|
+ Entries: 42
|
|
|
+ Dropped: 0
|
|
|
+
|
|
|
+ And here's an example that shows how to combine histogram data from
|
|
|
+ any two events even if they don't share any 'compatible' fields
|
|
|
+ other than 'hitcount' and 'stacktrace'. These commands create a
|
|
|
+ couple of triggers named 'bar' using those fields:
|
|
|
+
|
|
|
+ # echo 'hist:name=bar:key=stacktrace:val=hitcount' > \
|
|
|
+ /sys/kernel/debug/tracing/events/sched/sched_process_fork/trigger
|
|
|
+ # echo 'hist:name=bar:key=stacktrace:val=hitcount' > \
|
|
|
+ /sys/kernel/debug/tracing/events/net/netif_rx/trigger
|
|
|
+
|
|
|
+ And displaying the output of either shows some interesting if
|
|
|
+ somewhat confusing output:
|
|
|
+
|
|
|
+ # cat /sys/kernel/debug/tracing/events/sched/sched_process_fork/hist
|
|
|
+ # cat /sys/kernel/debug/tracing/events/net/netif_rx/hist
|
|
|
+
|
|
|
+ # event histogram
|
|
|
+ #
|
|
|
+ # trigger info: hist:name=bar:keys=stacktrace:vals=hitcount:sort=hitcount:size=2048 [active]
|
|
|
+ #
|
|
|
+
|
|
|
+ { stacktrace:
|
|
|
+ _do_fork+0x18e/0x330
|
|
|
+ kernel_thread+0x29/0x30
|
|
|
+ kthreadd+0x154/0x1b0
|
|
|
+ ret_from_fork+0x3f/0x70
|
|
|
+ } hitcount: 1
|
|
|
+ { stacktrace:
|
|
|
+ netif_rx_internal+0xb2/0xd0
|
|
|
+ netif_rx_ni+0x20/0x70
|
|
|
+ dev_loopback_xmit+0xaa/0xd0
|
|
|
+ ip_mc_output+0x126/0x240
|
|
|
+ ip_local_out_sk+0x31/0x40
|
|
|
+ igmp_send_report+0x1e9/0x230
|
|
|
+ igmp_timer_expire+0xe9/0x120
|
|
|
+ call_timer_fn+0x39/0xf0
|
|
|
+ run_timer_softirq+0x1e1/0x290
|
|
|
+ __do_softirq+0xfd/0x290
|
|
|
+ irq_exit+0x98/0xb0
|
|
|
+ smp_apic_timer_interrupt+0x4a/0x60
|
|
|
+ apic_timer_interrupt+0x6d/0x80
|
|
|
+ cpuidle_enter+0x17/0x20
|
|
|
+ call_cpuidle+0x3b/0x60
|
|
|
+ cpu_startup_entry+0x22d/0x310
|
|
|
+ } hitcount: 1
|
|
|
+ { stacktrace:
|
|
|
+ netif_rx_internal+0xb2/0xd0
|
|
|
+ netif_rx_ni+0x20/0x70
|
|
|
+ dev_loopback_xmit+0xaa/0xd0
|
|
|
+ ip_mc_output+0x17f/0x240
|
|
|
+ ip_local_out_sk+0x31/0x40
|
|
|
+ ip_send_skb+0x1a/0x50
|
|
|
+ udp_send_skb+0x13e/0x270
|
|
|
+ udp_sendmsg+0x2bf/0x980
|
|
|
+ inet_sendmsg+0x67/0xa0
|
|
|
+ sock_sendmsg+0x38/0x50
|
|
|
+ SYSC_sendto+0xef/0x170
|
|
|
+ SyS_sendto+0xe/0x10
|
|
|
+ entry_SYSCALL_64_fastpath+0x12/0x6a
|
|
|
+ } hitcount: 2
|
|
|
+ { stacktrace:
|
|
|
+ netif_rx_internal+0xb2/0xd0
|
|
|
+ netif_rx+0x1c/0x60
|
|
|
+ loopback_xmit+0x6c/0xb0
|
|
|
+ dev_hard_start_xmit+0x219/0x3a0
|
|
|
+ __dev_queue_xmit+0x415/0x4f0
|
|
|
+ dev_queue_xmit_sk+0x13/0x20
|
|
|
+ ip_finish_output2+0x237/0x340
|
|
|
+ ip_finish_output+0x113/0x1d0
|
|
|
+ ip_output+0x66/0xc0
|
|
|
+ ip_local_out_sk+0x31/0x40
|
|
|
+ ip_send_skb+0x1a/0x50
|
|
|
+ udp_send_skb+0x16d/0x270
|
|
|
+ udp_sendmsg+0x2bf/0x980
|
|
|
+ inet_sendmsg+0x67/0xa0
|
|
|
+ sock_sendmsg+0x38/0x50
|
|
|
+ ___sys_sendmsg+0x14e/0x270
|
|
|
+ } hitcount: 76
|
|
|
+ { stacktrace:
|
|
|
+ netif_rx_internal+0xb2/0xd0
|
|
|
+ netif_rx+0x1c/0x60
|
|
|
+ loopback_xmit+0x6c/0xb0
|
|
|
+ dev_hard_start_xmit+0x219/0x3a0
|
|
|
+ __dev_queue_xmit+0x415/0x4f0
|
|
|
+ dev_queue_xmit_sk+0x13/0x20
|
|
|
+ ip_finish_output2+0x237/0x340
|
|
|
+ ip_finish_output+0x113/0x1d0
|
|
|
+ ip_output+0x66/0xc0
|
|
|
+ ip_local_out_sk+0x31/0x40
|
|
|
+ ip_send_skb+0x1a/0x50
|
|
|
+ udp_send_skb+0x16d/0x270
|
|
|
+ udp_sendmsg+0x2bf/0x980
|
|
|
+ inet_sendmsg+0x67/0xa0
|
|
|
+ sock_sendmsg+0x38/0x50
|
|
|
+ ___sys_sendmsg+0x269/0x270
|
|
|
+ } hitcount: 77
|
|
|
+ { stacktrace:
|
|
|
+ netif_rx_internal+0xb2/0xd0
|
|
|
+ netif_rx+0x1c/0x60
|
|
|
+ loopback_xmit+0x6c/0xb0
|
|
|
+ dev_hard_start_xmit+0x219/0x3a0
|
|
|
+ __dev_queue_xmit+0x415/0x4f0
|
|
|
+ dev_queue_xmit_sk+0x13/0x20
|
|
|
+ ip_finish_output2+0x237/0x340
|
|
|
+ ip_finish_output+0x113/0x1d0
|
|
|
+ ip_output+0x66/0xc0
|
|
|
+ ip_local_out_sk+0x31/0x40
|
|
|
+ ip_send_skb+0x1a/0x50
|
|
|
+ udp_send_skb+0x16d/0x270
|
|
|
+ udp_sendmsg+0x2bf/0x980
|
|
|
+ inet_sendmsg+0x67/0xa0
|
|
|
+ sock_sendmsg+0x38/0x50
|
|
|
+ SYSC_sendto+0xef/0x170
|
|
|
+ } hitcount: 88
|
|
|
+ { stacktrace:
|
|
|
+ _do_fork+0x18e/0x330
|
|
|
+ SyS_clone+0x19/0x20
|
|
|
+ entry_SYSCALL_64_fastpath+0x12/0x6a
|
|
|
+ } hitcount: 244
|
|
|
+
|
|
|
+ Totals:
|
|
|
+ Hits: 489
|
|
|
+ Entries: 7
|
|
|
+ Dropped: 0
|