Overview

Strace can be used to output information about a session and is powerful tool for debugging and gaining more data about a process failure:

First thing is to locate the session PID, in our example we want to monitor a NAGIOS client, therefore we have to minitor a process called: “xinet” which is a parent to the nagios:

[root@oraback libexec]# ps -leaf | grep -i xinet
0 S root      4622 31523  0  78   0 - 15298 pipe_w 10:33 pts/0    00:00:00 grep -i xinet
1 S root      6177     1  0  75   0 -  5416 -      Jan19 ?        00:00:16 xinetd -stayalive -pidfile /var/run/xinetd.pid

Once we have located the parent process, we can start tracing:

[root@oraback libexec]# strace -o /tmp/x -fp 6177
Process 6177 attached - interrupt to quit
Process 4644 attached (waiting for parent)
Process 4644 resumed (parent 6177 ready)
Process 4645 attached (waiting for parent)
Process 4645 resumed (parent 4644 ready)
Process 4644 suspended
Process 4646 attached (waiting for parent)
Process 4646 resumed (parent 4645 ready)
Process 4647 attached (waiting for parent)

This command, will trace all info, into a file called: /tmp/x

In case of need, the trace can be extended to include more data using:

[root@oraback libexec]# strace -vo /tmp/x -fp 6177
Process 6177 attached - interrupt to quit
Process 5089 attached (waiting for parent)
Process 5089 resumed (parent 6177 ready)
Process 5091 attached (waiting for parent)
Process 5091 resumed (parent 5089 ready)
Process 5089 suspended
Process 5093 attached (waiting for parent)
Process 5093 resumed (parent 5091 ready)
Process 5093 detached
Process 5089 resumed
Process 5091 detached

After that we can examine the content of a trace file:

[root@oraback libexec]# less /tmp/x
2280  <... nanosleep resumed> NULL)     = 0
2280  nanosleep({1, 0},  <unfinished ...>
2286  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
2286  futex(0x38891f8, FUTEX_WAKE_PRIVATE, 1) = 0
2286  futex(0x3889224, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 161, {1528735731, 389058000}, ffffffff <unfinished ...>
2287  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
2279  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
2287  futex(0x38d5658, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
2279  futex(0x31f49f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
2287  <... futex resumed> )             = 0
2279  <... futex resumed> )             = 0
2287  futex(0x38d5684, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 319, {1528735726, 392983000}, ffffffff <unfinished ...>
2279  futex(0x31f4a24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 319, {1528735726, 392985000}, ffffffff <unfinished ...>
2278  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
2277  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
2275  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
2278  futex(0x31f4968, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
2277  futex(0x3884a78, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
2275  futex(0x31f4b18, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
2278  <... futex resumed> )             = 0
2277  <... futex resumed> )             = 0
2275  <... futex resumed> )             = 0
2278  futex(0x31f4994, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1589, {1528735722, 422523000}, ffffffff <unfinished ...>
2277  futex(0x3884aa4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1589, {1528735722, 422489000}, ffffffff <unfinished ...>