k8s_basic_storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
k8s_basic_storage [2020/05/22 15:34] – [Create PVC] andonovjk8s_basic_storage [2020/05/25 12:49] (current) – [Dynamic Provisioning] andonovj
Line 56: Line 56:
   storageClassName: ps-fast                    <- Indicate that it is in "ps-fast" storage Class   storageClassName: ps-fast                    <- Indicate that it is in "ps-fast" storage Class
   capacity:   capacity:
-    storage: 10Gi                              <- We allocate 10 GBs+    storage: 50Gi                              <- We allocate 50 GBs
   persistentVolumeReclaimPolicy: Retain           persistentVolumeReclaimPolicy: Retain        
 #                                   ^ #                                   ^
Line 85: Line 85:
 ubuntu@k8s-master:~/volume1$ kubectl get pv ubuntu@k8s-master:~/volume1$ kubectl get pv
 NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
-ps-pv   10Gi       RWO            Retain           Available           ps-fast                 3s+ps-pv   50Gi       RWO            Retain           Available           ps-fast                 3s
 ubuntu@k8s-master:~/volume1$ kubectl get pvc ubuntu@k8s-master:~/volume1$ kubectl get pvc
 No resources found in default namespace. No resources found in default namespace.
Line 109: Line 109:
   resources:   resources:
     requests:     requests:
-      storage: 10Gi+      storage: 50Gi
 </Code> </Code>
  
Line 119: Line 119:
 ubuntu@k8s-master:~/volume1$ kubectl get pvc ubuntu@k8s-master:~/volume1$ kubectl get pvc
 NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
-ps-pvc   Bound    ps-pv    10Gi       RWO            ps-fast        3s+ps-pvc   Bound    ps-pv    50Gi       RWO            ps-fast        3s
 ubuntu@k8s-master:~/volume1$ kubectl get pv ubuntu@k8s-master:~/volume1$ kubectl get pv
 NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE
-ps-pv   10Gi       RWO            Retain           Bound    default/ps-pvc   ps-fast                 5m8s+ps-pv   50Gi       RWO            Retain           Bound    default/ps-pvc   ps-fast                 5m8s
 ubuntu@k8s-master:~/volume1$ ubuntu@k8s-master:~/volume1$
 </Code> </Code>
  
 We see that, we have bounded them and we see the claim in the Persistent Volume description.  We see that, we have bounded them and we see the claim in the Persistent Volume description. 
-It is important to note that the PVC will bind to ANY PV which has the SAME or MORE of the requested storage. For example if the PVC wants to claim 10 GBs and we have PV with 20 GBs, it will bind. +It is important to note that the PVC will bind to ANY PV which has the SAME or MORE of the requested storage. For example if the PVC wants to claim 50 GBs and we have PV with 20 GBs, it will bind. 
  
-However, if our PVC wants to claim 20GBs and we have only 10 GB PV, then we are screwed and the PVC won't bind.+However, if our PVC wants to claim 20GBs and we have only 50 GB PV, then we are screwed and the PVC won't bind.
 Congrats, we have a pvc to present to any pod we want to have storage. Congrats, we have a pvc to present to any pod we want to have storage.
  
Line 138: Line 138:
  
 <Code:bash|Pod Specs> <Code:bash|Pod Specs>
 +apiVersion: v1
 kind: Pod kind: Pod
 metadata: metadata:
Line 143: Line 144:
 spec: spec:
   volumes:   volumes:
-    - name: fast10g+    - name: fast50g
       persistentVolumeClaim:       persistentVolumeClaim:
         claimName: ps-pvc         claimName: ps-pvc
Line 155: Line 156:
       volumeMounts:       volumeMounts:
       - mountPath: "/data"       - mountPath: "/data"
-        name: fast10g+        name: fast50g
 </Code> </Code>
  
Line 232: Line 233:
 root@node-1:/home/ubuntu/volume1# root@node-1:/home/ubuntu/volume1#
 </Code> </Code>
-Lo and behold, the simple text file is on persistent storage and won't be affected by the container anymore :)+Lo and behold, the simple text file is on persistent storage and won't be affected if the container crashes for example. It will stay there safe and sound on the host server.
  
-As we mentioned, that kind of persistent storage allocation DOESN'T SCALE, so let's see what scale :)+As we mentioned, that kind of persistent storage allocation DOESN'T SCALE and let's see why: 
 + 
 +{{ :kubernetes_storage_ps_yml_map.jpg?800 |}} 
 + 
 +You see that, mapping and mapping and mapping :) Let's see what we can do about it.
  
 =====Dynamic Provisioning===== =====Dynamic Provisioning=====
-Let's configure Dynamic storage.+Let's configure Dynamic storage. The idea here is that, we (as administrators) care only of the PVC, NOT the PV. So we create the PVC and the provisioner creates teh PV himself. 
 + 
 +Now, for Dynamic provisioning with NFS I had to re-configure the cluster. In a nutshell make sure that teh API IP which you give when you initiate the cluster has the same subnet of the pod network.  
 + 
 +For example: 
 + 
 +<Code:bash|Initiate Cluster for NFS> 
 +kubeadm init --ignore-preflight-errors=NumCPU --apiserver-advertise-address=192.168.50.10 --pod-network-cidr=192.168.50.0/24 
 +</Code> 
 + 
 +Calico by default is using: 192.168.0.0/16 so I modified it to: 192.168.50.0/24 so it will match the network of the API Advertise IP. 
 + 
 +So let's get going. In the begining, I had something like that: 
 + 
 +<Code:bash|Overview> 
 +NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE     IP                NODE         NOMINATED NODE   READINESS GATES 
 +kube-system   calico-kube-controllers-77c5fc8d7f-fcnh7   1/1     Running   0          3m56s   192.168.235.195   k8s-master   <none>           <none> 
 +kube-system   calico-node-94qkt                          1/1     Running   0          66s     192.168.50.12     node-2       <none>           <none> 
 +kube-system   calico-node-j54sq                          1/1     Running   0          2m18s   192.168.50.11     node-1       <none>           <none> 
 +kube-system   calico-node-rc4t6                          1/1     Running   0          3m56s   192.168.50.10     k8s-master   <none>           <none> 
 +kube-system   coredns-66bff467f8-d7hr5                   1/1     Running   0          10m     192.168.235.193   k8s-master   <none>           <none> 
 +kube-system   coredns-66bff467f8-jmwk7                   1/1     Running   0          10m     192.168.235.194   k8s-master   <none>           <none> 
 +kube-system   etcd-k8s-master                            1/1     Running   0          10m     192.168.50.10     k8s-master   <none>           <none> 
 +kube-system   kube-apiserver-k8s-master                  1/1     Running   0          10m     192.168.50.10     k8s-master   <none>           <none> 
 +kube-system   kube-controller-manager-k8s-master         1/1     Running   0          10m     192.168.50.10     k8s-master   <none>           <none> 
 +kube-system   kube-proxy-8td28                           1/1     Running   0          66s     192.168.50.12     node-2       <none>           <none> 
 +kube-system   kube-proxy-bljr8                           1/1     Running   0          10m     192.168.50.10     k8s-master   <none>           <none> 
 +kube-system   kube-proxy-dcnqt                           1/1     Running   0          2m18s   192.168.50.11     node-1       <none>           <none> 
 +kube-system   kube-scheduler-k8s-master                  1/1     Running   0          10m     192.168.50.10     k8s-master   <none>           <none> 
 +ubuntu@k8s-master:~$ 
 +</Code> 
 + 
 +So we have to create the following: 
 + 
 +  * Deployment 
 +  * Service Account 
 +  * Service 
 +  * RBAC 
 +  * Storage Class 
 +  * Claim 
 + 
 +====Create the Deployment,Service and Service Account==== 
 +You can see the deployment,service and service account's  YML below: 
 + 
 +<Code:bash|Components YML> 
 +apiVersion: v1 
 +kind: ServiceAccount 
 +metadata: 
 +  name: nfs-provisioner 
 +--- 
 +kind: Service 
 +apiVersion: v1 
 +metadata: 
 +  name: nfs-provisioner 
 +  labels: 
 +    app: nfs-provisioner 
 +spec: 
 +  ports: 
 +    - name: nfs 
 +      port: 2049 
 +    - name: nfs-udp 
 +      port: 2049 
 +      protocol: UDP 
 +    - name: nlockmgr 
 +      port: 32803 
 +    - name: nlockmgr-udp 
 +      port: 32803 
 +      protocol: UDP 
 +    - name: mountd 
 +      port: 20048 
 +    - name: mountd-udp 
 +      port: 20048 
 +      protocol: UDP 
 +    - name: rquotad 
 +      port: 875 
 +    - name: rquotad-udp 
 +      port: 875 
 +      protocol: UDP 
 +    - name: rpcbind 
 +      port: 111 
 +    - name: rpcbind-udp 
 +      port: 111 
 +      protocol: UDP 
 +    - name: statd 
 +      port: 662 
 +    - name: statd-udp 
 +      port: 662 
 +      protocol: UDP 
 +  selector: 
 +    app: nfs-provisioner 
 +--- 
 +kind: Deployment 
 +apiVersion: apps/v1 
 +metadata: 
 +  name: nfs-provisioner 
 +spec: 
 +  selector: 
 +    matchLabels: 
 +      app: nfs-provisioner 
 +  replicas: 1 
 +  strategy: 
 +    type: Recreate 
 +  template: 
 +    metadata: 
 +      labels: 
 +        app: nfs-provisioner 
 +    spec: 
 +      serviceAccount: nfs-provisioner 
 +      containers: 
 +        - name: nfs-provisioner 
 +          image: quay.io/kubernetes_incubator/nfs-provisioner:latest 
 +          ports: 
 +            - name: nfs 
 +              containerPort: 2049 
 +            - name: nfs-udp 
 +              containerPort: 2049 
 +              protocol: UDP 
 +            - name: nlockmgr 
 +              containerPort: 32803 
 +            - name: nlockmgr-udp 
 +              containerPort: 32803 
 +              protocol: UDP 
 +            - name: mountd 
 +              containerPort: 20048 
 +            - name: mountd-udp 
 +              containerPort: 20048 
 +              protocol: UDP 
 +            - name: rquotad 
 +              containerPort: 875 
 +            - name: rquotad-udp 
 +              containerPort: 875 
 +              protocol: UDP 
 +            - name: rpcbind 
 +              containerPort: 111 
 +            - name: rpcbind-udp 
 +              containerPort: 111 
 +              protocol: UDP 
 +            - name: statd 
 +              containerPort: 662 
 +            - name: statd-udp 
 +              containerPort: 662 
 +              protocol: UDP 
 +          securityContext: 
 +            capabilities: 
 +              add: 
 +                - DAC_READ_SEARCH 
 +                - SYS_RESOURCE 
 +          args: 
 +            - "-provisioner=example.com/nfs" 
 +          env: 
 +            - name: POD_IP 
 +              valueFrom: 
 +                fieldRef: 
 +                  fieldPath: status.podIP 
 +            - name: SERVICE_NAME 
 +              value: nfs-provisioner 
 +            - name: POD_NAMESPACE 
 +              valueFrom: 
 +                fieldRef: 
 +                  fieldPath: metadata.namespace 
 +          imagePullPolicy: "IfNotPresent" 
 +          volumeMounts: 
 +            - name: export-volume 
 +              mountPath: /export 
 +      volumes: 
 +        - name: export-volume 
 +          hostPath: 
 +            path: /srv 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ kubectl create -f deployment.yaml 
 +serviceaccount/nfs-provisioner created 
 +service/nfs-provisioner created 
 +deployment.apps/nfs-provisioner created 
 +</Code> 
 + 
 +Then let's create the RBAC, which created the Cluster roles and maps them and of course the storage class 
 + 
 +===Create RBAC and Storage Class=== 
 +<Code:bash|Create RBAC & Storage Class> 
 +kind: ClusterRole 
 +apiVersion: rbac.authorization.k8s.io/v1 
 +metadata: 
 +  name: nfs-provisioner-runner 
 +rules: 
 +  - apiGroups: [""] 
 +    resources: ["persistentvolumes"] 
 +    verbs: ["get", "list", "watch", "create", "delete"] 
 +  - apiGroups: [""] 
 +    resources: ["persistentvolumeclaims"] 
 +    verbs: ["get", "list", "watch", "update"] 
 +  - apiGroups: ["storage.k8s.io"] 
 +    resources: ["storageclasses"] 
 +    verbs: ["get", "list", "watch"] 
 +  - apiGroups: [""] 
 +    resources: ["events"] 
 +    verbs: ["create", "update", "patch"] 
 +  - apiGroups: [""] 
 +    resources: ["services", "endpoints"] 
 +    verbs: ["get"] 
 +  - apiGroups: ["extensions"] 
 +    resources: ["podsecuritypolicies"] 
 +    resourceNames: ["nfs-provisioner"] 
 +    verbs: ["use"] 
 +--- 
 +kind: ClusterRoleBinding 
 +apiVersion: rbac.authorization.k8s.io/v1 
 +metadata: 
 +  name: run-nfs-provisioner 
 +subjects: 
 +  - kind: ServiceAccount 
 +    name: nfs-provisioner 
 +     # replace with namespace where provisioner is deployed 
 +    namespace: default 
 +roleRef: 
 +  kind: ClusterRole 
 +  name: nfs-provisioner-runner 
 +  apiGroup: rbac.authorization.k8s.io 
 +--- 
 +kind: Role 
 +apiVersion: rbac.authorization.k8s.io/v1 
 +metadata: 
 +  name: leader-locking-nfs-provisioner 
 +rules: 
 +  - apiGroups: [""] 
 +    resources: ["endpoints"] 
 +    verbs: ["get", "list", "watch", "create", "update", "patch"] 
 +--- 
 +kind: RoleBinding 
 +apiVersion: rbac.authorization.k8s.io/v1 
 +metadata: 
 +  name: leader-locking-nfs-provisioner 
 +subjects: 
 +  - kind: ServiceAccount 
 +    name: nfs-provisioner 
 +    # replace with namespace where provisioner is deployed 
 +    namespace: default 
 +roleRef: 
 +  kind: Role 
 +  name: leader-locking-nfs-provisioner 
 +  apiGroup: rbac.authorization.k8s.io 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ cat class.yaml 
 +kind: StorageClass 
 +apiVersion: storage.k8s.io/v1 
 +metadata: 
 +  name: example-nfs 
 +provisioner: example.com/nfs 
 +mountOptions: 
 +  - vers=4.1 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ kubectl create -f rbac.yaml 
 +clusterrole.rbac.authorization.k8s.io/nfs-provisioner-runner created 
 +clusterrolebinding.rbac.authorization.k8s.io/run-nfs-provisioner created 
 +role.rbac.authorization.k8s.io/leader-locking-nfs-provisioner created 
 +rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-provisioner created 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ kubectl create -f class.yaml 
 +storageclass.storage.k8s.io/example-nfs created 
 +</Code> 
 + 
 +====Create Storage Claim==== 
 +With Dynamic provisioning, we DON'T Create the Volume, we create ONLY the claim, the volume is created automatically by the provision, that is the MAIN difference 
 + 
 +<Code:bash|Create Claim> 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ cat claim.yaml 
 +kind: PersistentVolumeClaim 
 +apiVersion: v1 
 +metadata: 
 +  name: nfs 
 +  annotations: 
 +    volume.beta.kubernetes.io/storage-class: "example-nfs" 
 +spec: 
 +  accessModes: 
 +    - ReadWriteMany 
 +  resources: 
 +    requests: 
 +      storage: 10Mi 
 +ubuntu@k8s-master:~/ 
 +</Code> 
 + 
 +====Verify==== 
 +We can verify the configuration as follows: 
 + 
 + 
 +<Code:bash|Create Claim> 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ kubectl get pv,pvc 
 +NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM         STORAGECLASS   REASON   AGE 
 +persistentvolume/pvc-9a8aa090-7c73-4e64-94eb-dcc7805828dd   10Mi       RWX            Delete           Bound    default/nfs   example-nfs             21m 
 + 
 +NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE 
 +persistentvolumeclaim/nfs   Bound    pvc-9a8aa090-7c73-4e64-94eb-dcc7805828dd   10Mi       RWX            example-nfs    21m 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ 
 +</Code> 
 + 
 +Finally, we have a bounded PVC using Dynamic Provision. There is one very good git with all this files: 
 + 
 +<Code:bash|Configure it with GIT> 
 +ubuntu@k8s-master:~$ git clone https://github.com/kubernetes-incubator/external-storage.git^C 
 +ubuntu@k8s-master:~$ 
 +ubuntu@k8s-master:~$ ls -lart 
 +total 32 
 +-rw-r--r--  1 ubuntu ubuntu 3771 Aug 31  2015 .bashrc 
 +-rw-r--r--  1 ubuntu ubuntu  220 Aug 31  2015 .bash_logout 
 +-rw-r--r--  1 ubuntu ubuntu  655 Jul 12  2019 .profile 
 +drwxr-xr-x  4 root   root   4096 May 25 10:51 .. 
 +drwx------  2 ubuntu ubuntu 4096 May 25 10:51 .ssh 
 +-rw-r--r--  1 ubuntu ubuntu    0 May 25 11:21 .sudo_as_admin_successful 
 +drwxrwxr-x  4 ubuntu ubuntu 4096 May 25 11:22 .kube 
 +drwxr-xr-x  5 ubuntu ubuntu 4096 May 25 11:27 . 
 +drwxrwxr-x 17 ubuntu ubuntu 4096 May 25 11:27 external-storage                         <---- This one 
 +</Code> 
 + 
 +====Create a Pod with Dynamic Provisioning==== 
 +We can of course create a pod which will be using the NFS, let's create NGINX pod for example: 
 + 
 +<Code:bash|Create NGINX Pod> 
 +apiVersion: apps/v1 
 +kind: Deployment 
 +metadata: 
 +  labels: 
 +    app: nginx 
 +  name: nfs-nginx 
 +spec: 
 +  replicas: 1 
 +  selector: 
 +    matchLabels: 
 +      app: nginx 
 +  template: 
 +    metadata: 
 +      labels: 
 +        app: nginx 
 +    spec: 
 +      volumes: 
 +      - name: nfs # 
 +        persistentVolumeClaim: 
 +          claimName: nfs  # same name of pvc that was created 
 +      containers: 
 +      - image: nginx 
 +        name: nginx 
 +        volumeMounts: 
 +        - name: nfs # name of volume should match claimName volume 
 +          mountPath: mydata2 # mount inside of contianer 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ kubectl create -f nginx.yml 
 +deployment.apps/nfs-nginx created 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ kubectl get pods -o wide 
 +NAME                              READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES 
 +nfs-nginx-6b4db6f57-4mczr         1/1     Running   0          2m51s   192.168.247.2   node-2   <none>           <none> 
 +nfs-provisioner-7795cf6f4-d7m2l   1/1     Running   0          67m     192.168.247.1   node-2   <none>           <none> 
 +ubuntu@k8s-master:~/external-storage/nfs/deploy/kubernetes$ 
 +</Code> 
 + 
 +Even an ubuntu pod: 
 +<Code:bash|ubuntu pod> 
 +ubuntu@k8s-master:~$ cat pod.yml 
 +apiVersion: v1 
 +kind: Pod 
 +metadata: 
 +  name: first-pod 
 +spec: 
 +  volumes: 
 +    - name: fast10m 
 +      persistentVolumeClaim: 
 +        claimName: nfs 
 +  containers: 
 +    - name: ctr1 
 +      image: ubuntu:latest 
 +      command: 
 +      - /bin/bash 
 +      - "-c" 
 +      - "sleep 60m" 
 +      volumeMounts: 
 +      - mountPath: "/data" 
 +        name: fast10m 
 +ubuntu@k8s-master:~$ kubectl get pods -o wide 
 +NAME                              READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES 
 +first-pod                         1/1     Running   0          24s     192.168.84.131   node-1   <none>           <none> 
 +nfs-nginx-6b4db6f57-4mczr         1/1     Running   0          4m16s   192.168.247.2    node-2   <none>           <none> 
 +nfs-provisioner-7795cf6f4-d7m2l   1/1     Running   0          69m     192.168.247.1    node-2   <none>           <none> 
 +ubuntu@k8s-master:~$ 
 +</Code> 
 + 
 +Eurika, finally we are done with both types of privisioning
  • k8s_basic_storage.1590161657.txt.gz
  • Last modified: 2020/05/22 15:34
  • by andonovj