When you make a secret in Kubernetes it’s not encrypted. It’s encoded in base64. This means anyone can decode your secret and log in as you. It also means you cannot commit your secret to a git repository unless you want others to have the ability to decode it.

The majority of secrets are meant to be secret. So this method is obviously not going to work for most production environments. Instead External Secrets Operator (ESO) exists. This dynamically pulls secrets from a provider, think of Hashicorp Vault, AWS Secrets Manager, or Google Cloud Secrets Manager, and dynamically creates a Kubernetes secret from it. This allows you to share your manifest without having to worry about accidentally leaking any passwords.

Another method is Sealed Secrets this works with asymmetric encryption - the public key is used to encrypt the secret and the private key, stored on the cluster, is used to decrypt it. This also solves the problem of secret management. Only the people who need to read it can. However there are a few issues with this. First passwords sometimes need to rotate, and most people don’t want the additional complexity of a CronJob to update the sealed secret. Secondly you’re still committing passwords to git. Which is just bad repository hygiene. Thirdly it’s not very production ready. Let’s say you get hacked, the pod has a ServiceAccount and the hacker is able to pull the decryption key, and now every sealed secret in the cluster is compromised.

For these reasons I went with the first option, External Secrets. There are lots of provider options for External Secrets, 1Password, Keeper Security, even ngrok. I went with Hashicorp Vault. This does have some added complexities but it was nice to have it run on cluster as I’m trying to have next to no costs for my homelab.

It’s relatively easy to pull secrets from Hashicorp Vault.

Secrets like LISA

First create your secret!

vault kv put secret/my-app/config \
    username="placeholder" \
    password="placeholder" \
    api_key="placeholder"

Then create your access policy

# policy.hcl
path "secret/data/my-app/config" {
  capabilities = ["read"]
}

vault policy write my-app-read policy.hcl

Then configure Vault’s Kubernetes auth

vault auth enable kubernetes # you only have to do this once

vault write auth/kubernetes/config \  # you only have to do this once
  kubernetes_host="https://kubernetes.default.svc:443" 

vault write auth/kubernetes/role/my-app-role \ # do this for each secret 
  bound_service_account_names="my-app-sa" \
  bound_service_account_namespaces="my-app" \
  policies="my-app-read" \
  ttl="1h"

Then create your SecretStore

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: my-app
spec:
  provider:
    vault:
      server: "http://hashicorp-vault.hashicorp-vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "my-app-role"
          serviceAccountRef:
            name: "my-app-sa"

And pull the secret!

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: my-app-secrets
  namespace: my-app
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: my-app-user-password
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: my-app/config
        property: username
    - secretKey: password
      remoteRef:
        key: my-app/config
        property: password

Congratulations! You just did secret management the production way. ESO automatically refreshes the secret every hour, so if you rotate it in Vault, the pod picks it up. No redeploys required.