Kubernetes Basics

Kubernetes (k8s) orchestrates containerized workloads — schedules them onto nodes, restarts crashed ones, scales them, exposes them, and rolls out updates without downtime. For a Python backend, “the service runs in pods that the cluster manages” replaces “I run gunicorn on an EC2 box and hope it stays up”.

The control plane and nodes

Control plane                    Worker nodes
+------------------+         +------------------+   +------------------+
| kube-apiserver   |         | kubelet          |   | kubelet          |
| etcd             |         | kube-proxy       |   | kube-proxy       |
| controller-mgr   |         | container runtime|   | container runtime|
| scheduler        |         |  (containerd)    |   |  (containerd)    |
+------------------+         | pods...          |   | pods...          |
                             +------------------+   +------------------+

• API server — the only thing you talk to. Everything flows through it.
• etcd — strongly consistent KV store; the cluster’s source of truth.
• scheduler — picks a node for each new pod.
• controller manager — runs the reconciliation loops (Deployment → ReplicaSet → Pod).
• kubelet — node-agent that runs containers via the runtime (containerd, CRI-O).
• kube-proxy — handles Service-to-pod routing on each node (iptables / IPVS / eBPF).

You don’t run any of this on EKS/GKE — the control plane is managed. You manage the nodes (or use Fargate / GKE Autopilot to skip that too).

The declarative model

You don’t tell k8s “create a pod.” You tell it “I want 3 replicas of this image always running.” A controller reconciles desired state (your YAML) with actual state (what’s running) on a loop. If a node dies, the controller notices, re-schedules pods elsewhere. If you kubectl delete pod, the controller spawns a replacement.

This is the single most important mental model: every k8s object is a declaration, and a controller is constantly reconciling. You don’t issue commands; you change the desired state and the system converges.

Core objects

kubectl — the CLI you actually use

kubectl get pods -n prod                       # list pods in namespace prod
kubectl describe pod orders-7d6 -n prod        # detail on one pod
kubectl logs orders-7d6 -n prod -f             # follow logs
kubectl exec -it orders-7d6 -n prod -- /bin/sh # shell inside
kubectl apply -f deployment.yaml               # declarative apply
kubectl rollout status deploy/orders -n prod   # watch a rollout
kubectl rollout undo deploy/orders -n prod     # rollback
kubectl port-forward svc/orders 8080:80 -n prod # tunnel to localhost
kubectl get events -n prod --sort-by=.lastTimestamp  # debug crashloops

The 90% rule: get, describe, logs, exec, apply. Everything else can wait.

A minimal Python service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  namespace: prod
spec:
  replicas: 3
  selector:
    matchLabels: { app: orders }
  template:
    metadata:
      labels: { app: orders }
    spec:
      containers:
      - name: app
        image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/orders:1.0.0
        ports: [{ containerPort: 8000 }]
        env:
        - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: orders-secrets, key: db_url }}}
        resources:
          requests: { cpu: "100m", memory: "256Mi" }
          limits:   { cpu: "500m", memory: "512Mi" }
        readinessProbe:
          httpGet: { path: /healthz/ready, port: 8000 }
          periodSeconds: 5
        livenessProbe:
          httpGet: { path: /healthz/live, port: 8000 }
          periodSeconds: 10

That’s a complete production-grade workload spec. Add a Service in front, an Ingress for external traffic, an HPA for autoscaling.

Resource requests vs limits

• requests — what the scheduler reserves; how it decides which node fits.
• limits — hard ceiling; CPU above limit gets throttled, memory above limit gets the pod OOMKilled.

Set both. Going requests-only means the scheduler can pack pods densely but they’ll fight at runtime. Going limits-only means k8s assumes worst case and wastes capacity.

Python gotcha: don’t set memory limits too tight on async services. Spikes during GC, large request payloads, or one bad query can OOMKill an otherwise healthy pod. Aim for limit ≈ 1.5–2× steady-state usage.

Namespaces

Logical grouping + isolation boundary for RBAC, quotas, and (with NetworkPolicies) traffic. Typical layout:

• prod, staging, dev per environment.
• Or orders, payments, users per team.

Pods in the same namespace can reach each other by short DNS name (orders resolves to orders.<ns>.svc.cluster.local). Cross-namespace needs the FQDN.

Labels and selectors

Everything is labeled; everything is selected by labels. Deployment selects pods via matchLabels: { app: orders }; Service selects pods the same way. Tags are arbitrary key/value pairs you control.

labels:
  app: orders
  version: v2
  team: payments

Selector mismatch is the #1 silent bug: Service exists, no endpoints, traffic 503s. kubectl get endpoints orders shows zero. Labels on pods don’t match Service selector.

What’s actually running where: pod scheduling

The scheduler picks a node based on:

• Resource requests — does it fit?
• Affinity / anti-affinity — “spread across AZs”, “co-locate with this”.
• Taints / tolerations — “GPU nodes only accept pods that tolerate the gpu taint”.
• Node selectors — “only nodes with label disk=ssd”.

Common patterns:

• Anti-affinity for HA — spread replicas across nodes/AZs so one node loss ≠ full outage.
• Taints on expensive nodes — GPU/ARM/spot nodes only run pods that explicitly tolerate them.

Failure modes & how to read them

• CrashLoopBackOff — container exits, k8s restarts, exits again. Check logs and exit code.
• ImagePullBackOff — image doesn’t exist or pull credentials wrong. Check registry, ECR perms.
• Pending — scheduler can’t place it. Check node resources, taints, pending PVCs.
• OOMKilled — exceeded memory limit. Bump limit or fix the leak.
• Evicted — node ran out of resources; the kubelet killed this pod to recover.

kubectl describe pod <name> shows the events at the bottom — start there.

Interview angle

• “What is a Pod?” — smallest deployable unit; one or more tightly-coupled containers sharing network namespace and volumes. Almost always one container per pod; sidecars (Envoy, log collectors) are the exception.
• “Deployment vs StatefulSet vs DaemonSet?” — Deployment for stateless replicas; StatefulSet for ordered, named, persistent-volume workloads (DBs); DaemonSet for one-per-node agents.
• “How does k8s know your pod is healthy?” — liveness probe (restart if unhealthy) and readiness probe (remove from Service rotation if not ready). Both required; liveness without readiness causes traffic to broken pods during startup.
• “What’s the difference between requests and limits?” — requests determine scheduling; limits enforce a runtime ceiling (CPU throttled, memory OOMKilled).
• “Why is k8s declarative?” — you state desired state; controllers continuously reconcile. Operations become “change the spec”; recovery from failures is automatic.