docker_advanced_k8s_troubleshooting [My wiki]

This is an old revision of the document!

Eventhough Kuberentes is pretty easy to configure, it is prone to problems, especailly if you do it on vagrant. So let's check which tools we can use, to monitor it:

Firstly we can get a component using the get command:

Get a component

ubuntu@k8s-master:~$ kubectl get pods
NAME                                     READY   STATUS             RESTARTS   AGE
busybox                                  0/1     Pending            0          13m
first-pod                                1/1     Running            7          21h
hello-deploy-7f44bd8b96-2xz8j            1/1     Running            2          20d
hello-deploy-7f44bd8b96-4c76j            1/1     Running            2          20d
hello-deploy-7f44bd8b96-7tvcs            1/1     Running            2          20d
hello-deploy-7f44bd8b96-9lnrm            1/1     Running            2          20d
hello-deploy-7f44bd8b96-dckq2            1/1     Running            2          20d
hello-deploy-7f44bd8b96-gnvwr            1/1     Running            2          20d
hello-deploy-7f44bd8b96-p66g8            1/1     Running            2          20d
hello-deploy-7f44bd8b96-qtxgk            1/1     Running            2          20d
hello-deploy-7f44bd8b96-qz6cr            1/1     Running            2          20d
hello-deploy-7f44bd8b96-r7g4q            1/1     Running            2          20d
nfs-client-provisioner-98cdf7875-26nbg   0/1     CrashLoopBackOff   7          18m
nfs-client-provisioner-98cdf7875-kdcvv   0/1     CrashLoopBackOff   7          18m

As you can see, we have 2 failing pods, but we don't know why. This is where the “describe” command comes to place:

Describe a resource

ubuntu@k8s-master:~$ kubectl describe pod nfs-client-provisioner-98cdf7875-26nbg
Name:         nfs-client-provisioner-98cdf7875-26nbg
Namespace:    default
Priority:     0
Node:         node-2/10.0.2.15
Start Time:   Sat, 23 May 2020 13:04:31 +0000
Labels:       app=nfs-client-provisioner
              pod-template-hash=98cdf7875
Annotations:  cni.projectcalico.org/podIP: 192.168.247.1/32
Status:       Running
IP:           192.168.247.1
IPs:
  IP:           192.168.247.1
Controlled By:  ReplicaSet/nfs-client-provisioner-98cdf7875
Containers:
  nfs-client-provisioner:
    Container ID:   docker://2e4c95d43caaef0bf2aae6400fe3eb349b8452501b04c8da494052843667e1d6
    Image:          quay.io/external_storage/nfs-client-provisioner:latest
    Image ID:       docker-pullable://quay.io/external_storage/nfs-client-provisioner@sha256:022ea0b0d69834b652a4c53655d78642ae23f0324309097be874fb58d09d2919
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Sat, 23 May 2020 13:19:21 +0000
      Finished:     Sat, 23 May 2020 13:19:51 +0000
    Ready:          False
    Restart Count:  7
    Environment:
      PROVISIONER_NAME:  example.com/nfs
      NFS_SERVER:        192.168.50.10
      NFS_PATH:          /srv/nfs/kubedata
    Mounts:
      /persistentvolumes from nfs-client-root (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from nfs-client-provisioner-token-ldqw7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  nfs-client-root:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    192.168.50.10
    Path:      /srv/nfs/kubedata
    ReadOnly:  false
  nfs-client-provisioner-token-ldqw7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nfs-client-provisioner-token-ldqw7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  18m                   default-scheduler  Successfully assigned default/nfs-client-provisioner-98cdf7875-26nbg to node-2
  Normal   Created    16m (x4 over 18m)     kubelet, node-2    Created container nfs-client-provisioner
  Normal   Started    16m (x4 over 18m)     kubelet, node-2    Started container nfs-client-provisioner
  Normal   Pulling    15m (x5 over 18m)     kubelet, node-2    Pulling image "quay.io/external_storage/nfs-client-provisioner:latest"
  Normal   Pulled     14m (x5 over 18m)     kubelet, node-2    Successfully pulled image "quay.io/external_storage/nfs-client-provisioner:latest"
  Warning  BackOff    3m19s (x52 over 17m)  kubelet, node-2    Back-off restarting failed container
ubuntu@k8s-master:~$

But even that, doesn't show us so much. So what, we can do. Well we can troubleshoot :)

Firstly, let's check the logs:

Check logs

ubuntu@k8s-master:~$ kubectl logs nfs-client-provisioner-98cdf7875-26nbg
Error from server (NotFound): the server could not find the requested resource ( pods/log nfs-client-provisioner-98cdf7875-26nbg)
ubuntu@k8s-master:~$

Well, that is specific issue to Vagrant configuration. The problem here is the fact that, the vagrant is using 10.0.2.15 IP as default IP, well that isn't the IP which you want for your kubernetes' API. So, the solution for that is to, edit the: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf, by adding the following:

Edit Kubelete config

Environment="KUBELET_EXTRA_ARGS=--node-ip=VAGRANT_VM_EXTERNAL_IP_HERE"

After that, we have to restart the kubelete using one of the following commands, (guess which one)

Kubelete Commands

systemctl stop kubelet
systemctl start kubelet
systemctl restart kubelet
systemctl status kubelet

But to be honest, better restart the whole Kubernetes:

Overview