docker_advanced_k8s_troubleshooting

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
docker_advanced_k8s_troubleshooting [2020/05/23 13:30] – andonovjdocker_advanced_k8s_troubleshooting [2020/05/25 12:47] (current) – [Not installed NFS Client (NFS Dynamic Provisioning)] andonovj
Line 121: Line 121:
 </Code> </Code>
  
-But to be honest, better restart the whole Kubernetes:+But to be honest, better restart the whole Kubernetes :) 
 +After that all should be running under the correct IPs:
  
 +<Code:bash|Check Kubernetes>
 +ubuntu@k8s-master:~$ kubectl get pods -o wide --all-namespaces
 +NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE     IP                NODE         NOMINATED NODE   READINESS GATES
 +kube-system   calico-kube-controllers-77c5fc8d7f-djzvd   1/1     Running   0          6m15s   192.168.235.195   k8s-master   <none>           <none>
 +kube-system   calico-node-f6nvk                          1/1     Running   0          6m15s   192.168.50.10     k8s-master   <none>           <none>
 +kube-system   calico-node-km6rz                          1/1     Running   0          3m53s   192.168.50.11     node-1       <none>           <none>
 +kube-system   calico-node-wgq4v                          1/1     Running   0          2m49s   192.168.50.12     node-2       <none>           <none>
 +kube-system   coredns-66bff467f8-5mntv                   1/1     Running   0          9m34s   192.168.235.194   k8s-master   <none>           <none>
 +kube-system   coredns-66bff467f8-6ks2w                   1/1     Running   0          9m34s   192.168.235.193   k8s-master   <none>           <none>
 +kube-system   etcd-k8s-master                            1/1     Running   0          9m44s   192.168.50.10     k8s-master   <none>           <none>
 +kube-system   kube-apiserver-k8s-master                  1/1     Running   0          9m44s   192.168.50.10     k8s-master   <none>           <none>
 +kube-system   kube-controller-manager-k8s-master         1/1     Running   0          9m44s   192.168.50.10     k8s-master   <none>           <none>
 +kube-system   kube-proxy-cbrt9                           1/1     Running   0          2m49s   192.168.50.12     node-2       <none>           <none>
 +kube-system   kube-proxy-lmn4d                           1/1     Running   0          3m53s   192.168.50.11     node-1       <none>           <none>
 +kube-system   kube-proxy-wfz74                           1/1     Running   0          9m34s   192.168.50.10     k8s-master   <none>           <none>
 +kube-system   kube-scheduler-k8s-master                  1/1     Running   0          9m44s   192.168.50.10     k8s-master   <none>           <none>
 +ubuntu@k8s-master:~$
 +ubuntu@k8s-master:~$ kubectl logs calico-node-km6rz --namespace kube-system
 +2020-05-25 07:49:24.728 [INFO][9] startup.go 256: Early log level set to info
 +2020-05-25 07:49:24.728 [INFO][9] startup.go 272: Using NODENAME environment for node name
 +2020-05-25 07:49:24.728 [INFO][9] startup.go 284: Determined node name: node-1
 +2020-05-25 07:49:24.729 [INFO][9] k8s.go 228: Using Calico IPAM
 +2020-05-25 07:49:24.729 [INFO][9] startup.go 316: Checking datastore connection
 +2020-05-25 07:49:24.737 [INFO][9] startup.go 340: Datastore connection verified
 +2020-05-25 07:49:24.737 [INFO][9] startup.go 95: Datastore is ready
 +2020-05-25 07:49:24.744 [INFO][9] startup.go 382: Initialize BGP data
 +2020-05-25 07:49:24.745 [INFO][9] startup.go 584: Using autodetected IPv4 address on interface enp0s8: 192.168.50.11/24
 +</Code>
 +
 +=====Specific Problems=====
 +Let's dwelves into more specific problems:
 +
 +====Wrong Api Version====
 +If you have a wrong Api version in teh YML file, you will receive the following error:
 +
 +<Code:bash|Wrong API Version>
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ kubectl create -f nginx.yml
 +error: unable to recognize "nginx.yml": no matches for kind "Deployment" in version "extensions/v1beta1"
 +</Code>
 +
 +To fix that, find where that image has been moved, most probably for newer versions of Kubernetes is:
 +
 +<Code:bash|Correct API Version for nginx>
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ cat nginx.yml
 +apiVersion: apps/v1
 +</Code>
 +
 +Bare in mind that can of course change.
 +
 +====Not installed NFS Client (NFS Dynamic Provisioning)====
 +In case you have NFS dynamic provisioning and you didn't install NFS client on ALL nodes of the cluster, you will get the following error, when you try to create a POD which is to be placed there.
 +
 +<Code:bash|Error with NFS client>
 +Mounting command: systemd-run
 +Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/64003b39-2316-4254-9e62-cb6d09f3fd6e/volumes/kubernetes.io~nfs/pvc-9a8aa090-7c73-4e64-94eb-dcc7805828dd --scope -- mount -t nfs -o vers=4.1 10.111.172.167:/export/pvc-9a8aa090-7c73-4e64-94eb-dcc7805828dd /var/lib/kubelet/pods/64003b39-2316-4254-9e62-cb6d09f3fd6e/volumes/kubernetes.io~nfs/pvc-9a8aa090-7c73-4e64-94eb-dcc7805828dd
 +Output: Running scope as unit run-r94910a4067104f079e9aae302b111ab2.scope.
 +mount: wrong fs type, bad option, bad superblock on 10.111.172.167:/export/pvc-9a8aa090-7c73-4e64-94eb-dcc7805828dd,
 +       missing codepage or helper program, or other error
 +       (for several filesystems (e.g. nfs, cifs) you might
 +       need a /sbin/mount.<type> helper program)
 +
 +       In some cases useful info is found in syslog - try
 +       dmesg | tail or so.
 +  Warning  FailedMount  42s  kubelet, node-1  MountVolume.SetUp failed for volume "pvc-9a8aa090-7c73-4e64-94eb-dcc7805828dd" : mount failed: exit status 32
 +</Code>
 +
 +To fix that, just install the NFS client on all the nodes:
 +
 +<Code:bash|Instgall NFS Client on Ubuntu>
 +root@node-1:~# apt-get install nfs-common
 +Reading package lists... Done
 +Building dependency tree
 +Reading state information... Done
 +The following additional packages will be installed:
 +  keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 python python-minimal python2.7 python2.7-minimal rpcbind
 +Suggested packages:
 +  watchdog python-doc python-tk python2.7-doc binfmt-support
 +The following NEW packages will be installed:
 +  keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 nfs-common python python-minimal python2.7 python2.7-minimal rpcbind
 +0 upgraded, 12 newly installed, 0 to remove and 40 not upgraded.
 +Need to get 4,258 kB of archives.
 +After this operation, 18.0 MB of additional disk space will be used.
 +Do you want to continue? [Y/n] Y
 +</Code>
 +
 +<Code:bash|Instgall NFS Client on Centos>
 +root@node-1:~# yum install -y nfs-utils
 +Reading package lists... Done
 +Building dependency tree
 +Reading state information... Done
 +The following additional packages will be installed:
 +  keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 python python-minimal python2.7 python2.7-minimal rpcbind
 +Suggested packages:
 +  watchdog python-doc python-tk python2.7-doc binfmt-support
 +The following NEW packages will be installed:
 +  keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 nfs-common python python-minimal python2.7 python2.7-minimal rpcbind
 +0 upgraded, 12 newly installed, 0 to remove and 40 not upgraded.
 +Need to get 4,258 kB of archives.
 +After this operation, 18.0 MB of additional disk space will be used.
 +Do you want to continue? [Y/n] Y
 +</Code>
  
  • docker_advanced_k8s_troubleshooting.1590240634.txt.gz
  • Last modified: 2020/05/23 13:30
  • by andonovj