This is an old revision of the document!
Overview
Just today (a specific time which I cannot tell) we had a time drif on the VM which most probably caused a FS to become a Read-Only. We don't know if the time drift caused the FS to become Read-Only, but we had to investigate anyway. There are couple command which you can use.
General
General trouble shooting isn't focused at specific component of the operation system, but it gives general idea of the system if there is performance issue, corruption and so on.
Logs
You can always check the logs. Very important log is:
/var/log/messages
That log records information about structural changes and other errors
Journals
You can also inspect the journals as follows:
Check journals since date
root@k8s-master:~# journalctl --since 2020-05-01 -- Logs begin at Fri 2020-05-01 13:35:29 UTC, end at Fri 2020-05-01 14:02:20 UTC. -- May 01 13:35:29 k8s-master systemd-journald[395]: Runtime journal (/run/log/journal/) is 1.2M, max 9.9M, 8.6M free. May 01 13:35:29 k8s-master kernel: Initializing cgroup subsys cpuset May 01 13:35:29 k8s-master kernel: Initializing cgroup subsys cpu May 01 13:35:29 k8s-master kernel: Initializing cgroup subsys cpuacct May 01 13:35:29 k8s-master kernel: Linux version 4.4.0-177-generic (buildd@lgw01-amd64-035) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12) ) #207-Ubuntu SMP Mon Mar 16 01:16:10 UTC 2020 (Ubuntu 4.4.0-177.2 May 01 13:35:29 k8s-master kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-177-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0 May 01 13:35:29 k8s-master kernel: KERNEL supported cpus: ************************************************************************
Commands
There are several commands, which can help you identify usage with the system:
- top (topas in AIX) - Shows you overall usage of the system: CPU, memory, swap and processes.
- lsof - Shows you a list of opened files.
- netent - Shows you established connections, for example: netstat -na | grep 1521, will show you all connects for port: 1521 which have been established or a service is listening for.
- e2fsck - BE CAREFUL with that command, usually it shouldn't be used very often as it can be dangerous if you don't know what you are doing. I general that command can check / fix corruption in a FS:
Check Corruption
[root@*********** log]# e2fsck -nv /dev/mapper/oravg-oradata e2fsck 1.42.9 (28-Dec-2013) Warning! /dev/mapper/oravg-oradata is mounted. Warning: skipping journal recovery because doing a read-only filesystem check. /dev/mapper/oravg-oradata contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong (12969949, counted=13514969). Fix? no Free inodes count wrong (3931129, counted=3931397). Fix? no 1031 inodes used (0.03%, out of 3932160) 398 non-contiguous files (38.6%) 2 non-contiguous directories (0.2%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 748/7 2758691 blocks used (17.54%, out of 15728640) 0 bad blocks 3 large files 742 regular files 12 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 754 files