How to Encrypt Kubernetes Secrets ?

Introduction

You’ve probably heard this not-so-secret secret many times – Kubernetes Secrets are not encrypted! The value of Secret is a base64 encoded string stored in etcd .

This means that anyone with access to your cluster can easily decode your sensitive data. anyone? Yes, almost anyone can, especially if the cluster’s RBAC is not set up correctly. Anyone can access the API or access etcd.

It could also be anyone who is authorized to create a pod or deploy in a namespace and then use that permission to retrieve all Secrets in that namespace. How to ensure that Secrets and other sensitive information (such as tokens) on the cluster are not leaked? In this blog post, we will discuss several ways to encrypt application Secrets when building, deploying, and running applications on K8s.

Secrets of K8s

Applications running on a Kubernetes cluster can use Kubernetes Secrets, which eliminates the need to store sensitive data such as tokens or passwords in application code.

The typical workflow of Secrets in the current default Kubernetes cluster is as follows:

1. Dev phase: Application developers using CICD use git as the source of truth for managing configurations deployed to the cluster. Access control helps secure access to the repository, but this by itself does not always ensure that an application’s sensitive information is not exposed.
2. Ops phase: The API server creates the Kubernetes Secrets resource on the cluster. You can read more about the Secrets lifecycle here . Secrets stored in etcd can be used by application pods in one of three ways:

1. Files mounted as volumes in one or more containers .
2. as container environment variables .
3. Used by the pod’s kubelet when pulling images .

In all three cases, the value in the ciphertext is decoded before use.

So, now that we know how it works, why isn’t just base64 encoding the ciphertext enough?

Why is Base64 encoding not considered ciphertext?

Base64 encoding is a binary-to-text encoding scheme that represents 24-bit binary data as 6-bit base64 numbers. It is used to transfer large amounts of data over the network, especially large files such as image files. Its primary function is to provide data integrity as it travels over the network . To be clear, encoding is not encryption.

Try this on any Linux terminal.

$ echo -n 'not encrypted' | base64 bm90IGVuY3J5cHRlZA== $ echo -n 'bm90IGVuY3J5cHRlZA==' | base64 --decode not encrypted SHELL

As above, anyone with access to your system can easily decode your Secrets, both when they are transferred to the cluster and when used on the cluster.

Here comes the problem

As a DevSecOps administrator, you obviously face two challenges:

1. How is sensitive data encrypted and managed outside the cluster, i.e. before entering the cluster during the build and deployment phases?
2. How do you keep sensitive data secure when running applications within a cluster?

The following are several options for encrypting K8s Secrets.

Encrypt secrets before deploying to cluster

As a developer pushing code to a git repository (aka the “source of truth” for your application), you can encrypt sensitive information used by your application before pushing your code to the git repository.

Using Bitnami Sealed Secrets

Bitnami Sealed Secrets Introduction

Bitnami Sealed Secrets: Kubernetes’ “ encrypted Secrets ”.

Typical usage scenarios:

Problem encountered : “I can manage all my K8s configurations in git except Secrets.”

Solution : Encrypt your Secret into a SealedSecret and store it securely even in public repositories. The SealedSecret can only be decrypted by the controller running in the target cluster, and no one else (even the original author) can obtain the original Secret from the SealedSecret.

Bitnami Sealed Secrets usage process

An example workflow using Bitnami Sealed Secrets is as follows:

1. The cluster administrator deploys the Sealed secrets controller on the K8s cluster
2. Developers need to install kubeseal CLI on their local computer.
3. The developer creates a Secret resource, and then kubeseal CLI obtains the key from the controller at runtime to encrypt or seal the resource. For network-constrained environments, the public key can also be stored locally and used by kubeseal.
   Kubeseal will create a SealedSecretcustom resource.
4. Developers push this CR to their own git repository
5. CR can be deployed on the cluster using CD tools such as ArgoCD.
6. The controller will detect the SealedSecret resource and decrypt it using the private key on the cluster.

Using KSOPS / Mozilla SOPS

Introduction to KSOPS/Mozilla SOPS

• Mozilla SOPS – sops is an editor for encrypted files, supports YAML, JSON, ENV, INI and BINARY formats and uses AWS KMS, GCP KMS, Azure Key Vault, age and PGP for encryption.
• KSOPS – A flexible Kustomize plugin for SOPS encrypted resources

KSOPS/Mozilla SOPS usage process

If you use Argo CD to deploy your application in Kubernetes, you can use the Kustomize SOPS plugin, which is used to decrypt resources encrypted using SOPS.

On the cluster, administrators will:

1. Deploy ArgoCD
2. Use age to generate keys
3. GitOpsCreate a key that stores the public and private keys in a specific (e.g. ) Namespace
4. Customize Argo CD to use Kustomize SOPS plug-in
5. Push the public key to the Git repository

Developers will:

1. Create Secret in local console
2. Use SOPS CLI to download the public key and encrypt the ciphertext
3. Use encrypted Secrets to generate KSOPS yaml and push it to the Git repository

ArgoCD uses KSOPS to decrypt secret files before deploying Secrets on the cluster.

summary

Both of the above methods are suitable for encrypting confidential files using asymmetric encryption technology. Both provide methods to decrypt sensitive data before it is deployed to the cluster as Secrets.

However, as you can see, once the encrypted data enters the cluster, it is decrypted before use. So this basically only solves part of the problem. Next, we need to look at how to secure this data in the cluster. Let’s look at the different options for encrypting data on a cluster.

Encrypting Secrets on a K8s cluster

etcd encryption options for K8s

By default, the K8s container platform does not encrypt etcd data. But native K8s, as well as some K8s distributions, provide the option to enable etcd-based encryption.

Here are some relevant reference documents:

1. Native K8s: Encrypting confidential data at rest | Kubernetes
2. OpenShift: Encrypting etcd data | Security and compliance | OpenShift Container Platform 4.13
3. K3s: Secret encryption | K3s

Readers can read further to learn more.

Use KMS driver for data encryption

In addition to the above etcd (static) encryption scheme, native K8s and some K8s distributions also provide (dynamic) data encryption schemes based on KMS drivers.

Here are some relevant reference documents:

1. Native K8s: Using KMS driver for data encryption | Kubernetes
2. GKE: Encrypt Secrets at the application layer | Google Kubernetes Engine (GKE) | Google Cloud
3. Amazon EKS: Enabling secret encryption on an existing cluster – Amazon EKS

Public Cloud/Private Cloud/Data Center Disk Encryption Options

Using EBS’s public cloud/private cloud/data center node-level encryption in K8s can provide an additional layer of encryption. Here we take public cloud as an example:

1. AWS: When hosting a K8s cluster on AWS, you can enable Amazon EBS encryption to provide encryption for EC2 instances. Amazon EBS encryption uses AWS KMS keys when creating encrypted volumes and snapshots. It uses AES-256-XTS for block cipher encryption. When you create an encrypted EBS volume and attach it to a supported instance type, the following types of data are encrypted:

• Data at rest within an encrypted volume
• All data moved between volumes and instances
• All snapshots created from encrypted volumes
• All volumes created from these snapshots

2. Azure: Provides encryption options for Azure Managed Disks connected to Azure Key Vault
3. Google provides encryption options for Google Cloud Storage . Both use AES 256 keys by default, but can also use customer-managed and provided keys and integrate with KMS.

Use third-party Secrets to store integrated Secrets

An important reason for choosing third-party Secrets storage is to ensure that the life cycle of Secrets is managed outside the cluster through a centralized Secrets storage solution. The authentication and authorization policies and procedures provided by these Secrets stores are different from those on the cluster and may be more suitable for controlling application data access.

Sidecar solution

Solutions such as Vault can be used to inject specific Secrets of an application pod. In this case, the sidecar/init container is responsible for authenticating to the Secret Provider, and the application can then use the returned Secrets if necessary. The connection to the Provider is via TLS to ensure the security of Secrets retrieval.

Customers who choose these solutions can decide to store secrets on or off the cluster. Typically, if customers have been using Vault for their infrastructure and other application needs, they will tend to integrate with these solutions for a seamless secrets management experience on K8s.

Secrets Storage CSI (SSCSI) Driver and Provider Solutions

The Secrets Store CSI driver allows Secrets and other sensitive information to be mounted as volumes into application pods. The Secrets Store CSI driver uses gRPC to communicate with the provider to retrieve Secrets content from the external Secrets Store specified in the SecretProviderClass custom resource. Once the volume is attached, the data within it is loaded into the container’s file system.

Unlike the sidecar solution described above, which pulls in Secrets content from a specific provider, the SSCSI driver can be configured to retrieve Secrets content from multiple different Secret Providers. For more information on how drivers and providers work, see here .

Customers who do not want to store secrets in etcd as Kubernetes secrets will primarily choose SSCSI for the following reasons

• They may have strict compliance requirements, making it necessary to store and manage secrets only in a central store rather than in a cluster.
• They may introduce workloads into an environment where the control plane is not managed by them, so they want full control over the confidentiality of the workload and don’t trust the platform administrator to do that. For example, customers bring workloads into tenants of a hosting provider’s cluster, or into a cloud platform whose control plane is not managed by them.

The SSCSI driver does not directly provide a way to protect non-volume mounted secrets, such as those that need to be pulled as environment variables or images, or those that you might create directly on the cluster to manage ingress certificates. However, you can use the sync secrets feature, which creates Kubernetes Secrets and then provides support for Secrets as Env variables.

External Secrets Operator (ESO)

External Secrets Operator (ESO) is a user-friendly solution for synchronizing secrets from external secrets management solutions into Kubernetes Secrets. ESO runs in the Kubernetes cluster as a deployment resource, uses Custom Resource Definitions (CRD) to configure access to Secret Provider through SecretStore resources, and uses ExternalSecret resources to manage Kubernetes secret resources.

Customers will choose ESO under the following circumstances:

• They need to integrate easily with the platform and be easy for developers to use
• They have a high level of trust in the cluster’s control plane – especially how etcd is configured for encryption or how RBAC is managed on the cluster.
• They have a multi-cluster use case for secrets management that requires cross-cluster secret integration
• They need to use management platform Secrets for non-applications, such as secrets for Ingress, automation, image pull
• Secrets need to be modified on the cluster and provide templates for specific applications
• Finally, and most importantly, their use case requires Secrets on the cluster

Integrate applications with HSM (Hardware Security Module)

The highest level of security, integrating applications or K8s with HSM (Hardware Security Module). Details omitted.

Conclusion

Today, we look at the various encryption options available with K8s and how each option protects sensitive data, so you can make an informed choice based on your use case and reality.

The following are some personal suggestions from the author, for reference only:

• Mainly using AWS, you can choose according to the security level: EBS encryption or KMS encryption
• If K8s is used in the data center and there are Secrets outside the K8s cluster that need to be managed, it is recommended to use Hashicorp Vault.
• If K8s is used in the data center and only K8s Secrets encryption is required, etcd static encryption and ESO are options that can be considered.

Check our last updated Kubernetes Exam Guides (CKAD , CKA , CKS) :