Kubernetes — Common Interview Questions and Answers

1. What problem does Kubernetes solve?

Schedules containerized workloads onto a fleet of nodes; restarts crashed ones; scales them up/down; exposes them via stable Services; rolls out new versions without downtime; reconciles a declarative spec automatically. It replaces “I run gunicorn on an EC2 box and hope” with “I declare what I want and the cluster maintains it.”

2. What’s a Pod?

The smallest deployable unit. One or more containers that share a network namespace, IP, and volumes; they start/restart/die together. Almost always one app container per pod. Multi-container pods are for sidecars (Envoy, log forwarders, OAuth2 proxies) or init containers (migrations).

3. Deployment vs StatefulSet vs DaemonSet?

• Deployment — stateless replicas. Rolling updates, scaling, no stable identity per pod.
• StatefulSet — stable pod names (pod-0, pod-1, …), stable per-pod persistent volumes, ordered start/stop. Use for DBs (postgres, kafka, elasticsearch).
• DaemonSet — one pod per node (or per node matching selector). Use for log forwarders, node-exporter, CNI agents.

4. Liveness vs Readiness probe — concrete failure for each?

• Liveness — “is the process alive?” Fail → restart container. Concrete: event loop is deadlocked. Don’t ping DB in liveness — DB blip would restart all pods.
• Readiness — “should this pod take traffic?” Fail → remove from Service endpoints (no restart). Concrete: DB pool can’t open, or migrations haven’t run.
• Startup — gates the others until it passes once; for slow-starting apps so liveness doesn’t kill them mid-boot.

5. Why is your Service returning 503?

Empty endpoints. Quick checks:

1. kubectl get endpoints <svc> — empty?
2. Service selector matches pod labels?
3. Pods are Ready (readiness passing)?
4. Service targetPort matches container port?

kubectl describe svc <name> shows the selector; kubectl get pods -l <selector> confirms matches.

6. How does a rolling update work?

Deployment creates a new ReplicaSet, scales it up while scaling the old one down, respecting:

• maxSurge — extra pods allowed during rollout.
• maxUnavailable — how many pods may be below target.

Each new pod must pass readiness before the next old pod is terminated. kubectl rollout status watches; kubectl rollout undo rolls back.

7. What’s the graceful shutdown sequence?

On SIGTERM (sent by k8s on pod termination):

1. Pod removed from Service endpoints (eventually — kube-proxy on every node must update iptables; ~seconds).
2. App receives SIGTERM → stop accepting new requests, drain in-flight, close pools, exit.
3. After terminationGracePeriodSeconds (default 30s), SIGKILL.

Classic bug: app exits before in-flight drains, or LB still routes during the iptables-propagation window. Fix: preStop: sleep 5 to wait for endpoint propagation, then drain.

8. ConfigMap vs Secret?

Same shape; semantic difference. ConfigMap for non-sensitive config; Secret for sensitive. But Secrets are base64, not encryption — anyone with get secrets perms can decode. For real safety: enable etcd encryption-at-rest + External Secrets Operator pulling from AWS Secrets Manager / Vault.

9. How do you get secrets into pods on AWS?

Several stacks; cleanest for EKS:

• IRSA (IAM Roles for Service Accounts) — pod’s ServiceAccount annotated with an IAM role ARN; AWS SDK assumes it via OIDC; pod calls secretsmanager.get_secret_value directly.
• External Secrets Operator — operator pulls from Secrets Manager and materializes a k8s Secret; pods consume normally; refresh on schedule.
• Secrets Store CSI Driver — mounts secrets as files directly from Secrets Manager / Vault, never persists in etcd.

Avoid: long-lived AWS access keys in env vars.

10. requests vs limits — when do you set each?

• requests — what the scheduler reserves. How it decides which node fits. HPA computes utilization vs requests.
• limits — runtime ceiling. CPU above limit gets throttled; memory above limit gets OOMKilled.

Always set both. Python gotcha: memory limit too tight + a transient spike or large request payload → OOMKilled. Aim for limit ≈ 1.5–2× steady state.

11. How do you autoscale a Celery worker?

HPA on an external metric — queue depth (SQS visible messages, Redis list length, Kafka consumer lag). KEDA is the de-facto operator that provides scalers for SQS / Redis / Kafka / Postgres / Prometheus without wiring custom adapters.

scaleTargetRef: { name: orders-worker }
metrics:
- type: External
  external:
    metric: { name: sqs_approximate_messages_visible }
    target: { type: AverageValue, averageValue: "50" }

“50 messages per pod” — backlog grows → scale up.

12. What’s an Ingress and how does it differ from a Service?

• Service — L3/L4. Stable ClusterIP for a set of pods. kube-proxy DNATs to a ready pod.
• Ingress — L7 (HTTP). Routes by host + path to backend Services. One LB can front many Services.

Ingress is just a spec; needs an Ingress Controller (nginx-ingress, AWS Load Balancer Controller, Traefik). Modern direction: Gateway API (k8s 1.29+ GA) replacing Ingress.

13. How do you do TLS termination?

cert-manager + Ingress. cert-manager requests certs from Let’s Encrypt / ACME / private CA, stores them as Secrets, the Ingress controller reads and terminates TLS at the LB. Annotation cert-manager.io/cluster-issuer: letsencrypt-prod on the Ingress triggers issuance/renewal automatically.

14. What does a NetworkPolicy default to?

Without any policy in a namespace, traffic is fully allowed between all pods. Once you create a NetworkPolicy selecting pod X, that pod becomes default-deny except for what the policy explicitly allows. Classic gotcha: forgetting to allow egress to DNS (53/UDP to kube-system) → everything breaks.

NetworkPolicies need a CNI that enforces them: Calico, Cilium, Weave. Flannel doesn’t.

15. Cluster Autoscaler vs Karpenter?

• Cluster Autoscaler — checks node groups (ASGs), scales them when pods are Pending.
• Karpenter (AWS, newer) — provisions nodes directly from EC2 based on Pending pods, picks instance types from a Provisioner/NodePool spec. Consolidates underutilized nodes automatically.

Karpenter is the modern default on EKS.

16. What’s IRSA?

IAM Roles for Service Accounts. EKS-specific. A pod’s ServiceAccount is annotated with an IAM role ARN. EKS configures an OIDC trust between the cluster and IAM. AWS SDKs inside the pod call AssumeRoleWithWebIdentity automatically, getting temporary STS creds for the role.

Result: no long-lived AWS access keys in the cluster; per-pod IAM least-privilege; rotation automatic.

17. How does k8s do service-to-service discovery?

CoreDNS + Service ClusterIP. From a pod, http://orders resolves to orders.<current-ns>.svc.cluster.local → Service ClusterIP. kube-proxy on each node DNATs to a ready pod via iptables/IPVS/eBPF.

You don’t write registry code on k8s; you create Services and let CoreDNS + kube-proxy handle it.

18. PodDisruptionBudget?

Limits how many pods can be voluntarily evicted at once (node drain, upgrade, cluster autoscaler scale-down). Prevents “drain a node → all 3 replicas gone briefly”.

spec:
  minAvailable: 2
  selector: { matchLabels: { app: orders } }

“At least 2 orders pods must be available during voluntary disruptions.” Doesn’t apply to involuntary (node crash, OOM). Pair with replica count and anti-affinity for real HA.

19. What’s a sidecar?

A second container in the same pod, sharing network and (optionally) volumes with the main app. Common sidecars:

• Envoy / linkerd-proxy — service mesh proxy.
• Log forwarder (fluent-bit) reading the app’s stdout or a shared volume.
• OAuth2 proxy for authentication.
• Cloud SQL proxy for managed DB authentication.

Native sidecar containers (k8s 1.29+) — initContainers with restartPolicy: Always. Older patterns: a regular container in the pod, with shared lifecycle gotchas.

20. Helm vs Kustomize?

• Helm — templated charts; package + version + parameterize; great for sharing (postgres, kafka, cert-manager charts). Indent gymnastics in templates.
• Kustomize — overlay/patch on raw YAML; built into kubectl apply -k; no templating.

Many teams: Helm for third-party charts, Kustomize for in-house apps. Or all-Kustomize via GitOps.

21. How do you do zero-downtime DB migrations on k8s?

Several layers:

1. Schema migration runs separately from app rollout. Helm pre-upgrade hook or Argo PreSync hook runs a Job before pods roll out.
2. Migration is backwards-compatible. Add column nullable → backfill async → deploy app that writes it → add NOT NULL constraint later.
3. Single migrator — leader election or advisory lock to avoid multiple pods racing on the migration.
4. App tolerates the schema flux during the rollout window.

Init containers for migrations is a beginner trap when you have multiple replicas — N pods race on the same migration.

22. What’s a StatefulSet’s headless service?

A Service with clusterIP: None. DNS returns each pod’s individual IP rather than a load-balanced VIP. Used with StatefulSets so clients can address specific pods (postgres-0.postgres, postgres-1.postgres) — necessary for primary/replica clusters.

23. Why is your pod Pending?

The scheduler can’t place it. Common reasons:

• No node has enough resources for the pod’s requests.
• Node selectors / affinity can’t be satisfied.
• Taints not tolerated (e.g., GPU node taint).
• Pending PVC — pod requested a volume, PV doesn’t exist, dynamic provisioner can’t create one.
• Image pull secrets missing for private registry.

kubectl describe pod <name> → Events at the bottom shows the actual reason.

24. What’s eviction?

The kubelet kills pods on a node that’s running out of resources (memory pressure, disk pressure) to recover. Pods with BestEffort QoS class get killed first; then Burstable; Guaranteed (requests == limits for all resources) is most protected.

Different from “termination” (k8s scheduling) — eviction is “this node is unhealthy”.

25. Common production gotchas?

• No anti-affinity — all replicas land on one node; node dies; full outage.
• No PodDisruptionBudget — node drain takes everything down briefly.
• Liveness pings the DB — DB blip cascades into pod restarts.
• terminationGracePeriodSeconds too short for slow shutdowns — SIGKILL during request handling → 502s.
• No resource requests — scheduler packs pods densely; pods fight at runtime; noisy neighbors.
• latest image tag — rollback is impossible because the tag moved.
• Secrets in git as plain YAML.
• No graceful shutdown — pods get SIGTERMed but ignore it; SIGKILL drops in-flight requests.

Mental model

Interview angle

• “What does Kubernetes actually give you?” - a declarative reconciliation loop: you state the desired state and controllers converge toward it, providing scheduling, self-healing, rolling updates and service discovery. The declarative model is the core idea.
• “Liveness or readiness probe?” - liveness failing restarts the container; readiness failing removes it from service endpoints without restarting. Putting a dependency check in liveness turns a slow database into a restart loop, which is the classic mistake.
• “How do you expose a service?” - ClusterIP internally, LoadBalancer for L4 external, and Gateway API for HTTP - which is GA and the recommended path now, with Ingress feature-frozen. See 08_gateway_api.md.
• “Requests versus limits?” - requests drive scheduling and guarantee capacity; limits cap usage. No memory limit lets one pod destabilise the node; a CPU limit that’s too low causes throttling that looks like application slowness.