Understanding Kubernetes Secrets: A Comprehensive Guide

Kubernetes Secrets are objects designed to store and manage sensitive information. By using Secrets, you can avoid embedding sensitive data directly in your application code or container images, thereby reducing the risk of accidental exposure. 

In this article, we will explore what Kubernetes secrets are, the types of Kubernetes Secrets, how to create and manage secrets, their use cases, best practices, its shortcomings, and alternatives.

Types of Kubernetes Secrets

Kubernetes provides several built-in types of Secrets to cater to common use cases. Each type has specific validations and constraints to ensure proper handling of the sensitive data.

1. Opaque Secrets

Opaque is the default Secret type if you don't specify a type. It is versatile and can store any kind of sensitive data.  It is used to store arbitrary user-defined data.

apiVersion: v1
kind: Secret
metadata:
  name: my-opaque-secret
type: Opaque
data:
  username: YWRtaW4=  # base64 encoded
  password: MWYyZDFlMmU2N2Rm 

Create using kubectl:

kubectl create secret generic my-opaque-secret \
  --from-literal=username=admin \
  --from-literal=password=1f2d1e2e67df

2. ServiceAccount Token Secrets

This type is used to provide long-lived ServiceAccount credentials to Pods. However, the recommended approach in Kubernetes v1.22 and later is to use the TokenRequest API for short-lived tokens. It is used to store a token credential that identifies a ServiceAccount.

apiVersion: v1
kind: Secret
metadata:
  name: my-sa-token-secret
  annotations:
    kubernetes.io/service-account.name: "my-service-account"
type: kubernetes.io/service-account-token
data:
  extra: YmFyCg==  # base64 encoded 

Auto-generated Legacy ServiceAccount Token Cleanup

Before v1.24, Kubernetes auto-generated Secret-based tokens for ServiceAccounts. These tokens were identified by a reference in the ServiceAccount's secrets field. From v1.29 onwards, unused auto-generated tokens are marked as invalid after one year and eventually purged.

apiVersion: v1
kind: Secret
metadata:
  name: build-robot-token
  namespace: default
  labels:
    kubernetes.io/legacy-token-last-used: 2024-09-13
    kubernetes.io/legacy-token-invalid-since: 2025-09-14
  annotations:
    kubernetes.io/service-account.name: build-robot
type: kubernetes.io/service-account-token

Create using kubectl:

kubectl create secret generic my-sa-token-secret \
  --type=kubernetes.io/service-account-token \
  --from-literal=extra=bar

3. Docker Config Secrets

There are two types of Docker config Secrets:

-  kubernetes.io/dockercfg: It stores a serialized `~/.dockercfg` file.

-  kubernetes.io/dockerconfigjson: It stores a serialized `~/.docker/config.json` file.

It is used to store credentials for accessing a container image registry.

apiVersion: v1
kind: Secret
metadata:
  name: my-docker-config-secret
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2V4YW1wbGUvdjEvIjp7ImF1dGgiOiJvcGVuc2VzYW1lIn19fQo=  # base64 encoded JSON

Create using kubectl:

This is particularly useful when you don't have a pre-existing Docker configuration file. 

kubectl create secret docker-registry my-docker-config-secret \
  --docker-server=my-registry.example.com \
  --docker-username=my-username \
  --docker-password=my-password \
  --docker-email=my-email@example.com

This command creates a Secret of type kubernetes.io/dockerconfigjson and stores the Docker registry credentials.

‍
4. Basic Authentication Secrets

This type is specifically for storing a username and password for basic authentication. It is used to store credentials for basic authentication.

apiVersion: v1
kind: Secret
metadata:
  name: my-basic-auth-secret
type: kubernetes.io/basic-auth
stringData:
  username: admin
  password: t0p-Secretstore

Create using kubectl:

kubectl create secret generic my-basic-auth-secret \
  --type=kubernetes.io/basic-auth \
  --from-literal=username=admin \
  --from-literal=password=t0p-Secretstore

5. SSH Authentication Secrets

This type is used to store SSH private keys for authentication purposes.

apiVersion: v1
kind: Secret
metadata:
  name: my-ssh-auth-secret
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: UG91cmluZzYlRW1vdGljb24lU2N1YmE=  # base64 encoded private key

Create using kubectl:

kubectl create secret generic my-ssh-auth-secret \
  --type=kubernetes.io/ssh-auth \
  --from-file=ssh-privatekey=path/to/private/key

6. TLS Secrets

This type is commonly used to configure encryption in transit for an Ingress. It is used to store a certificate and its associated key for TLS.

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
type: kubernetes.io/tls
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNVakNDQWJzQ0FnMytNQTBHQ1NxR1NJYjNEUUVCQlFVQU1JR2JNUXN3Q1FZRFZRUUdFd0pLVURFT01Bd0cKQTFVRUNCTUZWRzlyZVc4eEVEQU9CZ05WQkFjVEIwTm9kVzh0YTNVeEVUQVBCZ05WQkFvVENFWnlZVzVyTkVSRQpNUmd3RmdZRFZRUUxFdzlYWldKRFpYSjBJRk4xY0hCdmNuUXhHREFXQmdOVkJBTVREMFp5WVc1ck5FUkVJRmRsCllpQkRRVEVqTUNFR0NTcUdTSWIzRFFFSkFSWVVjM1Z3Y0c5eWRFQm1jbUZ1YXpSa1pDNWpiMjB3SGhjTk1UTXcKTVRFeE1EUTFNVE01V2hjTk1UZ3dNVEV3TURRMU1UTTVXakJMTVFzd0NRWURWUVFHREFKS1VERVB...
  tls.key: RXhhbXBsZSBkYXRhIGZvciB0aGUgVExTIGNydCBmaWVsZA==  # base64 encoded private key

Create using kubectl

kubectl create secret tls tls-secret \
 --cert=path/to/cert/file \
 --key=path/to/key/file

7. Bootstrap Token Secrets

This type is used for tokens that sign well-known ConfigMaps during the bootstrap process. It is used to store tokens used during the node bootstrap process.

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-6emitej
  namespace: kube-system
type: bootstrap.kubernetes.io/token
data:
  auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4=
  expiration: MjAyMC0wOS0xM1QwNDozOToxMFo=
  token-id: NWVtaXRq
  token-secret: a3E0Z2lodnN6emduMXAwcg==
  usage-bootstrap-authentication: dHJ1ZQ==
  usage-bootstrap-signing: dHJ1ZQ==

Create using kubectl:

kubectl create secret generic bootstrap-token-6emitej \
  --type=bootstrap.kubernetes.io/token \
  --from-literal=auth-extra-groups=system:bootstrappers:kubeadm:default-node-token \
  --from-literal=expiration=2020-09-13T04:39:10Z \
  --from-literal=token-id=5emitej \
  --from-literal=token-secret=kq4gihvszzgn1p0r \
  --from-literal=usage-bootstrap-authentication=true \
  --from-literal=usage-bootstrap-signing=true

Creating and Managing Secrets

As we've already shown, one can use kubectl to create various types of Secrets. While this method is convenient for development and testing, it is not recommended for production environments due to security concerns. Instead, consider using manifest files, kustomize tool that provides better security and version control.

Using Secrets in Pods

Secrets can be used in Pods either as environment variables or as volume mounts.

‍

As Environment Variables: To use a Secret in an environment variable, add an environment variable for each Secret key in the env[].valueFrom.secretKeyRef field.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: myimage
    env:
    - name: USERNAME
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: username
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: password

As Volume Mounts: Secrets can be mounted as data volumes in a Pod. Kubernetes ensures that the specified object reference points to an object of type Secret. 

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: myimage
    volumeMounts:
    - name: secret-volume
      mountPath: "/etc/secret-volume"
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

There are other ways to use secrets in a Pod. These are:

Optional Secrets: You can mark a Secret as optional. If an optional Secret doesn't exist, Kubernetes ignores it.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: nginx
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret
      optional: true

Container Image Pull Secrets: To fetch container images from a private repository, you can configure image pull Secrets.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: myprivateimage
  imagePullSecrets:
  - name: my-docker-config-secret

Immutable Secrets: Kubernetes allows you to mark Secrets as immutable, preventing any changes to the data. This improves cluster performance and protects against accidental updates

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQ=
immutable: true

Use Cases for Kubernetes Secrets

Use Case 1: Pod with SSH Keys

To securely manage SSH keys within a Kubernetes cluster, you can create a Secret that contains your SSH private and public keys. This is useful for applications that require SSH access to other systems.

kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub

You can reference the SSH key Secret in a Pod and consume it as a volume:

apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod
  labels:
    name: secret-test
spec:
  volumes:
    - name: secret-volume
      secret:
        secretName: ssh-key-secret
  containers:
    - name: ssh-test-container
      image: mySshImage
      volumeMounts:
        - name: secret-volume
          readOnly: true
          mountPath: "/etc/secret-volume"

When the container runs, the SSH keys will be available at: /etc/secret-volume/ssh-publickey,  /etc/secret-volume/ssh-privatekey The container can then use these keys to establish SSH connections securely.

Use Case 2: Pods with Production and Test Credentials

In environments where different credentials are required for production and test environments, you can create separate Secrets for each:

kubectl create secret generic prod-db-mysecret --from-literal=username=produser --from-literal=password=Y4nys7f11
kubectl create secret generic test-db-mysecret --from-literal=username=testuser --from-literal=password=iluvtests

You can create Pods that consume these Secrets, ensuring that each environment has the appropriate credentials:

apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Pod
    metadata:
      name: prod-db-client-pod
      labels:
        name: prod-db-client
    spec:
      volumes:
        - name: secret-volume
          secret:
            secretName: prod-db-mysecret
      containers:
        - name: db-client-container
          image: myClientImage
          volumeMounts:
            - name: secret-volume
              readOnly: true
              mountPath: "/etc/secret-volume"
  - apiVersion: v1
    kind: Pod
    metadata:
      name: test-db-client-pod
      labels:
        name: test-db-client
    spec:
      volumes:
        - name: secret-volume
          secret:
            secretName: test-db-mysecret
      containers:
        - name: db-client-container
          image: myClientImage
          volumeMounts:
            - name: secret-volume
              readOnly: true
              mountPath: "/etc/secret-volume"

Both containers will have the following files present on their filesystems: /etc/secret-volume/username/, etc/secret-volume/password. This approach allows you to maintain a common Pod configuration template, differing only in the Secret used.

Use Case 3: Dotfiles in Secret Volume

To make a piece of data "hidden" (i.e., in a file whose name begins with a dot character), you can define a key that begins with a dot:

apiVersion: v1
kind: Secret
metadata:
  name: dotfile-mysecret
data:
  .secret-file: dmFsdWUtMg0KDQo=

Mount the Secret into a volume in a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: secret-dotfiles-pod
spec:
  volumes:
    - name: secret-volume
      secret:
        secretName: dotfile-mysecret
  containers:
    - name: dotfile-test-container
      image: registry.k8s.io/busybox
      command: [ "ls", "-l", "/etc/secret-volume" ]
      volumeMounts:
        - name: secret-volume
          readOnly: true
          mountPath: "/etc/secret-volume"

The secret-volume will contain a single file called .secret-file, and the container will have this file present at /etc/secret-volume/.secret-file.

Note: Files beginning with dot characters are hidden from the output of ls -l; use ls -la to see them when listing directory contents.

Best Practices Using Secrets:

1. Configure Encryption at Rest: By default, Secret objects are stored unencrypted in etcd. To enhance security, configure encryption for your Secret data in etcd. This ensures that even if someone gains access to etcd, they cannot easily read the sensitive information. Read more here.

2.  Implement Least-Privilege Access: When setting up access control mechanisms like Kubernetes Role-Based Access Control (RBAC), adhere to the principle of least privilege:

Components: Restrict watch or list access to only the most privileged, system-level components. Grant get access for Secrets only if necessary for the component's normal operation.

Humans: Limit get, watch, or list access to Secrets. Only cluster administrators should have access to etcd, including read-only access. For more granular control, consider third-party authorization mechanisms.

3. Use Short-Lived Secrets: To minimize the risk of exposure, use short-lived Secrets that are rotated frequently. This reduces the window of opportunity for an attacker to exploit a compromised Secret.

4. Implement Audit Rules: Set up audit rules to alert on specific events, such as the concurrent reading of multiple Secrets by a single user. This helps in detecting and responding to security breaches.

5. Improve etcd Management Policies:

Data Wiping: Consider wiping or shredding the durable storage used by etcd once it is no longer in use.

Encrypted Communication: If there are multiple etcd instances, configure encrypted SSL/TLS communication between them to protect Secret data in transit.

7. Configure Access to External Secrets: Utilize third-party Secret store providers to keep your confidential data outside the cluster. The Kubernetes Secrets Store CSI Driver allows the kubelet to retrieve Secrets from external stores and mount them as volumes into specific Pods. You can check here the providers for the Secret Store CSI Driver.

Shortcomings of Kubernetes secrets and alternatives:

While Kubernetes Secrets are a common way to handle confidential data, they come with certain limitations. For example, Kubernetes Secrets are mounted to Pods unencrypted and are stored unencrypted in etcd, which can pose security risks. To overcome these limitations, several alternative methods can provide enhanced security and flexibility. Let's explore:

1. ServiceAccounts and Tokens: When your cloud-native component needs to authenticate to another application within the same Kubernetes cluster, using a ServiceAccount and its associated tokens can be an effective approach. ServiceAccounts are Kubernetes objects that provide an identity for processes running in a Pod. By assigning a ServiceAccount to a Pod, you can use Kubernetes' built-in authentication mechanisms to securely identify and authorize your client.

2. Third-Party Tools for Managing Sensitive Data: There are several third-party tools designed to manage sensitive data securely. These tools can run either within or outside your Kubernetes cluster and provide a secure way to handle secrets. For example, you can use a service that Pods access over HTTPS, which reveals a secret only if the client correctly authenticates using a ServiceAccount token.
Popular tools include:

HashiCorp Vault: A tool for securely accessing secrets. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log.

AWS Secrets Manager: A service to manage secrets used by your applications, services, and IT resources.

3. Custom Signers for X.509 Certificates: For authentication purposes, you can implement a custom signer for X.509 certificates. By using CertificateSigningRequests (CSRs), you can have your custom signer issue certificates to Pods that need them. This method uses the robust security properties of X.509 certificates and can be useful for mutual TLS (mTLS) authentication between services.

4. Device Plugins for Node-Local Encryption Hardware: In scenarios where you need to leverage hardware-based security features, you can use device plugins to expose node-local encryption hardware to specific Pods. For example, you can schedule trusted Pods onto nodes equipped with a Trusted Platform Module (TPM), which is configured out-of-band.

5. Combining Multiple Methods: In many cases, the best approach is to combine multiple methods to achieve a layered security model. For example, you can deploy an operator that fetches short-lived session tokens from an external service and then creates Kubernetes Secrets based on those tokens. The operator ensures that the tokens are valid and refreshed as needed, while the Pods use the Secrets without needing to know the underlying mechanisms.

Always remember that security is a continuous process. Regularly review and update your security practices to adapt to new threats and vulnerabilities. By doing so, you can ensure that your Kubernetes cluster remains secure and resilient against potential attacks.

Secrets aren't the most secure but they are most efficient performance-wise. And if your cluster performance is important - try PerfectScale to monitor and optimize Kubernetes performance, cost and reliability.
Get optimized your cluster today. Book a demo now with PerfectScale team!