Kubernetes secrets

A Kubernetes secret is an object that stores sensitive authentication information like passwords, tokens, or keys. A secret is either auto-generated by the control plane and populated as a service-account-token or created manually by a user. Anyone with root-level access to a cluster has access to the "secrets" for that cluster. It is a best practice to rotate secrets on a regular basis or otherwise as-needed (i.e. an employee leaves a company or role, general security practices, etc.). This guide reviews the steps required to rotate the auto-generated service-account-token secrets in kube-system using kubectl, as well as how to rotate your lke-admin-token via the Linode CLI.

Note that the instructions in this guide only apply to service-account-token secrets in kube-system; depending on your application configuration, there may be additional user-generated secrets in other namespaces you may want to delete or rotate manually.

Before you begin

You will need to have the kubectl command line tool installed and your local system configured to communicate with your cluster. To do this, you must download your cluster's kubeconfig file and save its file path to your $KUBECONFIG environment variable. Instructions can be found here: Access and Download your kubeconfig

Rotate your cluster's secrets

Rotate service-account-token secrets in the kube-system namespace

Once your local $KUBECONFIG environment variable is configured with your cluster's kubeconfig file, follow these steps to rotate the auto-generated service-account-token secrets within your cluster.

List the secrets in your cluster's kube-system namespace and filter by service-account-token to see secrets auto-generated by the control plane:

kubectl -n kube-system get secrets --field-selector="type=kubernetes.io/service-account-token"

You should see output similar to the following:

NAME                                        TYPE                                  DATA   AGE
ccm-user-token-wk56d                        kubernetes.io/service-account-token   3      3m12s
cluster-autoscaler-user-token-klhfn         kubernetes.io/service-account-token   3      78s
kube-state-metrics-user-token-mxfft         kubernetes.io/service-account-token   3      77s
kubernetes-dashboard-lke-user-token-8jgkx   kubernetes.io/service-account-token   3      77s
lke-admin-token-lxjv6                       kubernetes.io/service-account-token   3      3m2s

Delete the secrets.

kubectl -n kube-system delete secrets --field-selector="type=kubernetes.io/service-account-token"

You should receive a confirmation the secrets have been deleted:

secret "ccm-user-token-wk56d" deleted
secret "cluster-autoscaler-user-token-klhfn" deleted
secret "kube-state-metrics-user-token-mxfft" deleted
secret "kubernetes-dashboard-lke-user-token-8jgkx" deleted
secret "lke-admin-token-lxjv6" deleted

Once your service-account-token secrets have been deleted, you will no longer have access to your cluster using your existing kubeconfig. To regain access, you will need to regenerate your kubeconfig and re-export your local environment variable using the steps below in Regenerate Your Kubeconfig.

Regenerate your kubeconfig

In order to expedite secret recreation after deletion and regain access to your cluster, you can regenerate your kubeconfig. This process will also regenerate the lke-admin-token secret in kube-system. Note that these steps use the Linode CLI and follow the Kubernetes Cluster Regenerate instructions from our API documentation. See our API documentation for correlating API commands.

View your list of Kubernetes clusters to get the ID number of the cluster you wish to target:

linode-cli lke clusters-list

The cluster ID can be found in the id column:

┌────────┬─────────────────┬────────┬─────────────┬─────────────────────────────────┐
│ id     │ label           │ region │ k8s_version │ control_plane.high_availability │
├────────┼─────────────────┼────────┼─────────────┼─────────────────────────────────┤
│ 12345  │ example-cluster │ us-iad │ 1.31        │ True                            │
└────────┴─────────────────┴────────┴─────────────┴─────────────────────────────────┘

Regenerate your cluster's kubeconfig file using the cluster ID obtained in step 1. Replace {{< placeholder "12345" >}} with your cluster's ID number:
```
linode-cli lke regenerate {{< placeholder "12345" >}} --kubeconfig=true --servicetoken=true
```
Download your new kubeconfig file from Cloud Manager by navigating to the Kubernetes section, clicking on your cluster's more options ellipsis, and selecting Download Kubeconfig. It may take a few minutes for the new kubeconfig file to become available.

📘
Alternatively, you can use the Linode CLI to view your new kubeconfig and overwrite your existing kubeconfig.yaml file, replacing {{< placeholder "12345" >}} with your cluster ID and {{< placeholder "~/Downloads/kubeconfig.yaml" >}} with the file path to your current kubeconfig:
linode-cli --json lke kubeconfig-view 12345 | jq -r '.[].kubeconfig' | base64 -d > ~/Downloads/kubeconfig.yaml
If you choose to download your new kubeconfig via Cloud Manager, you will need to reconfigure your kubeconfig by saving the file path to your $KUBECONFIG environment variable. Replace {{< placeholder "~/Downloads/kubeconfig.yaml" >}} with the file path for your new kubeconfig file:
```
export KUBECONFIG={{< placeholder "~/Downloads/kubeconfig.yaml" >}}
```
Verify that you have access to your cluster by viewing your nodes:
```
kubectl get nodes
```
Delete your old kubeconfig file once you have verified access to your cluster.
Now that a new kubeconfig has been generated and configured, your service-account-token secrets and lke-admin-token secret have been rotated.

Additional best practices

This guide reviews steps for rotating secrets in the kube-system namespace used by the lke-managed control plane. As a best practice, it is recommended to use separate namespaces for any application-specific service-account-tokens and secrets.

Depending on your specific use case and configuration, there may be additional security steps you wish to take for your application-specific service-account-tokens once your secrets have been rotated, including restarting pods or recycling nodes.

Restart your pods

View your pods:
```
kubectl get pods
```
Restart your pods:
```
kubectl rollout restart
```

Recycle your nodes

When recycling a worker node, the corresponding Compute Instance is deleted and replaced with a new one. Nodes can be recycled via Cloud Manager using the instructions in our Manage Nodes and Node Pools guide here: Recycle Nodes