linux_config_troubleshooting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
linux_config_troubleshooting [2020/05/01 14:15] – created andonovjlinux_config_troubleshooting [2020/05/01 14:19] (current) – andonovj
Line 75: Line 75:
 ------------ ------------
          754 files          754 files
 +</Code>
 +
 +=====Specific=====
 +You can choose to trace a specific process and what commands to the kernel he is passing as follows:
 +
 +====Using Strace====
 +Strace is a tool which can be installed and it is used to display what request the application is passing to the kernel to execute. Usually these commands are very low level: "Read pointer", "Write data", "Allocate latch" and etc.
 +
 +===Locate a session===
 +First thing is to locate the session PID, in our example we want to monitor a NAGIOS client, therefore we have to minitor a process called: "xinet" which is a parent to the nagios:
 +
 +<sxh bash>
 +[root@oraback libexec]# ps -leaf | grep -i xinet
 +0 S root      4622 31523  0  78   0 - 15298 pipe_w 10:33 pts/0    00:00:00 grep -i xinet
 +1 S root      6177     1  0  75   0 -  5416 -      Jan19 ?        00:00:16 xinetd -stayalive -pidfile /var/run/xinetd.pid
 +</sxh>
 +
 +===Trace===
 +Once we have located the parent process, we can start tracing:
 +
 +
 +<Code:bash|Start tracing>
 +[root@oraback libexec]# strace -o /tmp/x -fp 6177
 +Process 6177 attached - interrupt to quit
 +Process 4644 attached (waiting for parent)
 +Process 4644 resumed (parent 6177 ready)
 +Process 4645 attached (waiting for parent)
 +Process 4645 resumed (parent 4644 ready)
 +Process 4644 suspended
 +Process 4646 attached (waiting for parent)
 +Process 4646 resumed (parent 4645 ready)
 +Process 4647 attached (waiting for parent)
 +</Code>
 +
 +This command, will trace all info, into a file called: /tmp/x
 +
 +===Verbose===
 +In case of need, the trace can be extended to include more data using:
 +
 +
 +<Code:bash|Enable Verbose mode>
 +[root@oraback libexec]# strace -vo /tmp/x -fp 6177
 +Process 6177 attached - interrupt to quit
 +Process 5089 attached (waiting for parent)
 +Process 5089 resumed (parent 6177 ready)
 +Process 5091 attached (waiting for parent)
 +Process 5091 resumed (parent 5089 ready)
 +Process 5089 suspended
 +Process 5093 attached (waiting for parent)
 +Process 5093 resumed (parent 5091 ready)
 +Process 5093 detached
 +Process 5089 resumed
 +Process 5091 detached
 +</Code>
 +
 +After that we can examine the content of a trace file:
 +
 +
 +<Code:bash|Check the log>
 +[root@oraback libexec]# less /tmp/x
 +2280  <... nanosleep resumed> NULL)     = 0
 +2280  nanosleep({1, 0},  <unfinished ...>
 +2286  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
 +2286  futex(0x38891f8, FUTEX_WAKE_PRIVATE, 1) = 0
 +2286  futex(0x3889224, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 161, {1528735731, 389058000}, ffffffff <unfinished ...>
 +2287  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
 +2279  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
 +2287  futex(0x38d5658, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
 +2279  futex(0x31f49f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
 +2287  <... futex resumed> )             = 0
 +2279  <... futex resumed> )             = 0
 +2287  futex(0x38d5684, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 319, {1528735726, 392983000}, ffffffff <unfinished ...>
 +2279  futex(0x31f4a24, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 319, {1528735726, 392985000}, ffffffff <unfinished ...>
 +2278  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
 +2277  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
 +2275  <... futex resumed> )             = -1 ETIMEDOUT (Connection timed out)
 +2278  futex(0x31f4968, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
 +2277  futex(0x3884a78, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
 +2275  futex(0x31f4b18, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
 +2278  <... futex resumed> )             = 0
 +2277  <... futex resumed> )             = 0
 +2275  <... futex resumed> )             = 0
 +2278  futex(0x31f4994, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1589, {1528735722, 422523000}, ffffffff <unfinished ...>
 +2277  futex(0x3884aa4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1589, {1528735722, 422489000}, ffffffff <unfinished ...>
 </Code> </Code>
  • linux_config_troubleshooting.1588342525.txt.gz
  • Last modified: 2020/05/01 14:15
  • by andonovj