Draining a node

You can use kubectl drain to safely evict all of the pods from a node before you perform maintenance on the node such as kernel upgrade, hardware maintenance, and others. Safe evictions allow the containers of the pods to gracefully terminate and respect the PodDisruptionBudgets that you specified. For more information see, Disruptions.

Kubernetes workloads move around the cluster, which enables use cases like highly available distributed systems. Linode recommends you to move any data storage on the filesystem of the Compute Instances in an LKE cluster to Persistent Volumes with network attached storage. Avoid using local storage on LKE nodes whenever possible. If you are using Persistent Volume Claim for the application on an LKE cluster, skip the entire Copy the application data to a Persistent Volume section and proceed directly to Add a new node pool to the cluster and drain the nodes.

This guide provides instructions to:

Copy the application data to a Persistent Volume if you are using a local storage to store application data.
Add a new node pool to the cluster and then drain the nodes.

Before you begin

This guide assumes you have a working Linode Kubernetes Engine (LKE) cluster running on Linode and you are familiar with PodDisruptionBudget concept and Configured PodDisruptionBudgets for applications that need them.

Install the Kubernetes CLI (kubectl) on the local computer.
Follow the instructions in Deploying and Managing a Cluster with Linode Kubernetes Engine Tutorial to connect to an LKE cluster.

📘
Ensure that the KUBECONFIG context is persistent
Ensure that Kubernetes CLI is using the right cluster context. Run the get-contexts subcommand to check:
```
kubectl config get-contexts
```

Copy the application data to a persistent volume

❗️
The instructions in this section creates a Block Storage volume billable resource on your Linode account. A single volume can range from 10 GB to 10,000 GB in size and costs $0.10/GB per month or $0.00015/GB per hour. If you do not want to keep using the Block Storage volume that you create, be sure to delete it when you have finished the guide.
If you remove the resources afterward, you are only billed for the hour(s) that the resources were present on your account. Consult the Billing and Payments guide for detailed information about how hourly billing works and for a table of plan pricing.

Create a Persistent Volume Claim (PVC) that consumes a Block Storage volume. To create a PVC, create a manifest file with the following YAML:
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: linode-block-storage-retain
```
📘
To retain the Block Storage volume and its data, even after the associated PVC is deleted, use the linode-block-storage-retain StorageClass. If, instead, you prefer to have the Block Storage volume and its data deleted along with its PVC, use the linode-block-storage StorageClass. For more information, see the Delete a Persistent Volume Claim.

The PVC represents a Block Storage volume. Because Block Storage volumes have a minimum size of 10 gigabytes, the storage has been set to 10Gi. If you choose a size smaller than 10 gigabytes, the PVC defaults to 10 gigabytes. Currently the only mode supported by the Linode Block Storage CSI driver is ReadWriteOnce, meaning that it can only be connected to one Kubernetes node at a time.
Create the PVC in Kubernetes, and pass in the pvc.yaml file:
```
kubectl create -f pvc.yaml
```
After a few moments the Block Storage volume is provisioned and the Persistent Volume Claim is ready to use.

Check the status of the PVC by typing the following command:

kubectl get pvc

An output similar to the following appears:

NAME          STATUS     VOLUME                 CAPACITY     ACCESS MODES   STORAGECLASS                  AGE
pvc-test      Bound      pvc-0e95b811652111e9    10Gi         RWO           linode-block-storage-retain   2m

You can now attach the PVC to a Pod.

Create a manifest file for the new Pod using the following YAML, where application is using local storage at $MOUNTPATH, pvc-test is a Persistent Volume Claim at $CSIVolumePath:

apiVersion: v1
kind: Pod
metadata:
  name: new-pod
  labels:
    app: application
      volumes:
      - name: application
        hostPath:
          path: $HOSTPATH
      - name: pvc-test
        persistentVolumeClaim:
          claimName: pvc-test
    ........
      volumeMounts:
      - name: application
        mountPath: $MOUNTPATH
      - name: pvc-test
        mountPath: $CSIVolumePath

Create a new Pod named new-pod:
```
kubectl create -f new-pod.yaml
```
After a few moments the Pod should be up and running. To check the status of the Pod, type the following command:
```
kubectl get pods
```
An output similar to following appears:
```
NAME       READY   STATUS    RESTARTS   AGE
new-pod   1/1     Running   0          2m
```
Connect to a shell in the new Pod, type the following command:
```
kubectl exec -it new-pod -- /bin/bash
```
From the shell, copy the files from local storage to the PVC. In the following command $MOUNTPATH is the location of the local storage and $CSIVolumePath is the location on the PVC:
```
cp -P $MOUNTPATH $CSIVolumePath.
```
Delete the new Pod that you created, and then re-create it:
```
kubectl delete pod new-pod
kubectl create -f new-pod.yaml
```
You should now see that all the data is stored in the CSI Volume.

Add a new node to the cluster and drain the node

Add an additional Node Pool to the LKE cluster, of a plan type and size which can accommodate the existing workloads.
After the new Compute Instances have joined the cluster, drain any Compute Instance that are scheduled for maintenance. This causes the workloads to be rescheduled to other Compute Instances in the cluster. Linode recommends draining one Compute Instance at a time to ensure that the workloads have been rescheduled to other Compute Instances. An example Node drain command:
```
kubectl drain lke9297-11573-5f3e357cb447
```
You can delete the old Node Pool or choose to keep it for after the maintenance is complete. Note, if you keep the Node Pool, you will be charged for it.
When the maintenance has been completed and if you kept your previous Compute Instances, after they have booted you can mark them as scheduled again by using the following command:
```
kubectl uncordon lke9297-11573-5f3e357cb447
```