backend / kubernetes / 03_configmaps_and_secrets.md

ConfigMaps and Secrets

5 interview angles 5 min read source

ConfigMaps and Secrets

How your app gets its config and secrets at runtime. ConfigMap for non-sensitive; Secret for sensitive — though stock Secrets are barely better than ConfigMaps (base64, not encryption).

ConfigMap

Non-secret config: feature flags, URLs of external services, log levels, tuning parameters.

apiVersion: v1
kind: ConfigMap
metadata: { name: orders-config, namespace: prod }
data:
  LOG_LEVEL: "info"
  FEATURE_NEW_CHECKOUT: "true"
  application.toml: |
    [database]
    pool_size = 20
    timeout = 5

Two ways to consume:

As env vars

spec:
  containers:
  - name: app
    envFrom:
    - configMapRef: { name: orders-config }
    # or specific keys:
    env:
    - name: LOG_LEVEL
      valueFrom: { configMapKeyRef: { name: orders-config, key: LOG_LEVEL } }

Env vars are convenient but immutable for the life of the pod. Changing the ConfigMap doesn’t update env in running pods — you must restart.

As mounted files

spec:
  containers:
  - name: app
    volumeMounts:
    - { name: config, mountPath: /etc/app, readOnly: true }
  volumes:
  - name: config
    configMap: { name: orders-config }

Files at /etc/app/application.toml etc. Mounted file content updates when the ConfigMap changes (with ~minute lag), but env-style consumption does not. If your app re-reads the file periodically, you can hot-reload config.

Secret

apiVersion: v1
kind: Secret
metadata: { name: orders-secrets, namespace: prod }
type: Opaque
data:
  db_password: cGFzc3dvcmQxMjM=    # base64; NOT encryption
stringData:
  db_url: "postgres://user:password123@db/orders"

Consumed identically (envFrom: secretRef, secretKeyRef, file mount).

Why stock Secrets are weak

  • base64, not encryption. Anyone with kubectl get secret -o yaml can decode.
  • At rest in etcd in plaintext unless you configure Encryption at Rest with KMS.
  • Visible to anyone who can exec into a pod (env vars / mounted files).

The real-world fix: external secret stores

Bring secrets from a managed store (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) into pods without committing them to git or storing them in etcd plaintext.

External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: orders-secrets, namespace: prod }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-secrets, kind: ClusterSecretStore }
  target: { name: orders-secrets }
  data:
  - secretKey: db_password
    remoteRef: { key: prod/orders, property: db_password }
  - secretKey: api_key
    remoteRef: { key: prod/orders, property: stripe_api_key }

The operator pulls from AWS Secrets Manager and materializes a regular k8s Secret. Pods consume that Secret normally. Rotation happens transparently when the operator refreshes.

Secrets Store CSI Driver

Mount secrets directly from Vault / AWS / Azure / GCP as files, without ever materializing them as k8s Secrets:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: { name: orders }
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/orders"
        objectType: "secretsmanager"
volumes:
- name: secrets-store
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes: { secretProviderClass: "orders" }

Stronger: secrets never touch etcd at all.

AWS-specific: IRSA + Secrets Manager

The cleanest pattern on EKS — IAM Roles for Service Accounts (IRSA). The pod’s ServiceAccount is mapped to an IAM role; AWS SDKs inside the pod assume it automatically (temporary STS creds via OIDC).

apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/orders-pod-role
# inside the pod — just use boto3, no creds in env
import boto3
client = boto3.client("secretsmanager")
secret = client.get_secret_value(SecretId="prod/orders")
db_url = json.loads(secret["SecretString"])["db_url"]

Or use [External Secrets Operator] with IRSA on the operator pod itself. No long-lived AWS keys in the cluster.

At-rest encryption for etcd

If you must use native Secrets, configure encryption at rest with KMS:

# kube-apiserver config — managed by EKS / GKE, you set it via cluster config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: ["secrets"]
  providers:
  - kms:
      name: aws-kms-provider
      endpoint: ...
  - identity: {}

EKS has a checkbox for this; GKE auto-encrypts; self-hosted is up to you.

RBAC and access

Anyone with get secrets permission can read all Secret values. Tight RBAC matters:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { namespace: prod, name: orders-read }
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
# NOTE: not granting "secrets"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { namespace: prod, name: orders-read }
subjects: [{ kind: ServiceAccount, name: orders, namespace: prod }]
roleRef: { kind: Role, name: orders-read, apiGroup: rbac.authorization.k8s.io }

Granting humans cluster-admin is the default antipattern. Use namespace-scoped roles + groups in your OIDC IdP.

Common bugs

  • Updated ConfigMap, app still uses old value. Env-var consumption is fixed at pod start; restart the pod (kubectl rollout restart deploy/orders) or use a mounted file + hot-reload.
  • Secret in git. Don’t commit Secret YAML with real values. Use Sealed Secrets, External Secrets, SOPS, or Vault.
  • stringData and data collision. Both populate the same data; stringData wins. Picking one consistently helps.
  • Mounted Secret as env var with multi-line content. Container env doesn’t handle newlines well; use file mount.
  • Forgetting optional: true. Pod fails to start if a missing optional ConfigMap is referenced.

Interview angle

  • “How do you get secrets into your pods?” — for real production: External Secrets Operator or CSI driver pulling from AWS Secrets Manager / Vault. Native k8s Secrets are base64, not encryption, and ideally combined with etcd encryption-at-rest.
  • “What’s wrong with putting a password in a ConfigMap?” — ConfigMaps aren’t secrets — they appear in plaintext in logs, audit, kubectl describe. Use a Secret (or better, an external store).
  • “What’s IRSA?” — IAM Roles for Service Accounts. EKS-specific. Pod’s ServiceAccount is annotated with an IAM role ARN; the AWS SDK inside the pod assumes that role via OIDC. No long-lived creds in the cluster.
  • “How do you rotate secrets without downtime?” — external store (Vault/Secrets Manager) rotates the value; External Secrets Operator refreshes the k8s Secret on its interval; app re-reads on schedule or on next pool reset. For DB passwords specifically: dual-password support during rotation window.
  • “Does updating a ConfigMap update running pods?” — files mounted from a ConfigMap update (with lag, ~minute); env vars consumed from a ConfigMap do NOT update — pod restart required.
Previous
Pods, Services, Deployments