backend / cloud aws / containers / README.md

Amazon EKS (Elastic Kubernetes Service)

5 min read index source

Amazon EKS (Elastic Kubernetes Service)

Managed Kubernetes control plane on AWS. AWS runs the K8s API server, scheduler, controller manager, and etcd; you bring (or let AWS bring) the worker nodes. For K8s fundamentals see backend/17_kubernetes/.

Control plane vs nodes

  • Control plane — fully managed by AWS, costs $0.10/hour (~$73/month) per cluster.
  • Nodes — your responsibility. Options:
    • Self-managed EC2 — full control, you patch them.
    • Managed Node Groups — AWS provisions EC2, you pick instance types; AWS does patching with rolling drain.
    • Fargate — serverless pods, no nodes to manage. Pay per pod vCPU/RAM.

Most production clusters mix Managed Node Groups (steady workloads on Reserved/Spot instances) + Fargate (burst, low-volume).

IRSA — IAM Roles for Service Accounts

The EKS-specific feature that matters most for security: pods get IAM identities via Kubernetes ServiceAccounts, no long-lived AWS keys in the cluster.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: orders
  namespace: prod
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/orders-pod-role
spec:
  serviceAccountName: orders
  containers:
  - name: app
    image: ...

Then in code:

import boto3
s3 = boto3.client("s3")
s3.put_object(...)        # uses orders-pod-role automatically

The IAM role’s trust policy lists the cluster OIDC + the ServiceAccount; AWS SDK calls AssumeRoleWithWebIdentity using the projected token from the pod. Per-pod least-privilege without sidecars.

Successor: EKS Pod Identity (2023+) — a slightly simpler API, but IRSA is still the dominant pattern.

Karpenter — the modern autoscaler

Where you’d use Cluster Autoscaler on most K8s, EKS users prefer Karpenter:

  • Provisions EC2 nodes directly from EC2 API (no Auto Scaling Groups).
  • Picks instance type per pending pod, from a NodePool spec of allowed shapes.
  • Consolidates underutilized nodes automatically.
  • Bin-packs aggressively — better utilization than ASG-based autoscaling.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      requirements:
      - { key: kubernetes.io/arch,     operator: In, values: [amd64, arm64] }
      - { key: karpenter.sh/capacity-type, operator: In, values: [on-demand, spot] }
      - { key: node.kubernetes.io/instance-type, operator: In, values: [c6g.large, c6g.xlarge, m6g.large, m6g.xlarge] }
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

Karpenter mixes spot + on-demand, ARM + x86, and consolidates as load drops. Big cost win on spiky workloads.

VPC CNI and networking

EKS uses AWS VPC CNI by default — each pod gets a real VPC IP from the same subnet as the node. Pods are first-class VPC citizens; can use VPC Security Groups directly (with the Security Groups for Pods feature).

Implication: instance type caps pod density (each instance has limited ENI/IP allocation). Use Prefix Delegation to get more pods per node.

Alternative CNIs: Cilium (eBPF), Calico (NetworkPolicy + advanced security). Most teams stay on AWS VPC CNI unless they need NetworkPolicy or eBPF features.

Load balancing

AWS Load Balancer Controller

Watches Ingress objects and Service: LoadBalancer; provisions ALBs / NLBs accordingly. Most efficient pattern:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip       # bypass NodePort, direct to pod IP
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
spec:
  rules: ...

target-type: ip registers pod IPs directly with the ALB target group (no NodePort hop). Use it.

Ingress vs Gateway API

Gateway API (k8s 1.29+ GA) is the more expressive successor. EKS supports it via the AWS Gateway API Controller. New deployments — prefer Gateway API for richer routing.

Storage

  • EBS CSI driver — block storage for StatefulSets.
  • EFS CSI driver — shared file system; multi-pod RWX.
  • FSx CSI — high-performance.

EBS is single-AZ; pin pods to nodes in the same AZ as the volume (via topology constraints).

Cluster security: control plane + addons

  • EKS Auto Mode (2024+) — AWS manages nodes + addons (Karpenter, VPC CNI, AWS Load Balancer Controller, CoreDNS, kube-proxy). Closest to “K8s without ops.”
  • EKS Add-ons — managed installs of core components. Use these instead of self-hosting.
  • Pod Security Standards — enforce baseline / restricted profiles via Pod Security admission.
  • GuardDuty for EKS — runtime threat detection.

EKS Fargate

Pods scheduled on AWS-managed compute, no nodes you maintain.

apiVersion: eksctl.io/v1alpha5
kind: FargateProfile
metadata: { name: orders-fp }
spec:
  cluster: prod
  selectors:
  - namespace: prod
    labels: { compute: fargate }

Pods in prod namespace with label compute: fargate run on Fargate; others on EC2 nodes. Costs more per CPU/RAM than EC2 (especially Spot), but no node ops.

Mix and match: dev/CI on Fargate (no node management); steady production on Karpenter + Spot for cost.

Common gotchas

  • No autoscaler in a fresh cluster. Pods Pending → no nodes added. Install Karpenter or Cluster Autoscaler explicitly.
  • VPC CNI IP exhaustion at high pod density on smaller instance types. Enable Prefix Delegation.
  • Cluster upgrades. Control plane is one click; node groups must be drained and replaced. Run two K8s versions side-by-side during upgrade.
  • CoreDNS at scale can become a bottleneck. Scale up replicas; use NodeLocal DNSCache for high-QPS workloads.
  • aws-auth ConfigMap legacy: IAM users mapped to K8s users via a single ConfigMap. Easy to break cluster auth. EKS Access Entries (newer) is the API-driven replacement.

When to pick EKS over ECS

  • You already use K8s (multi-cloud, in-house tooling, hires).
  • You want polyglot orchestration with a portable mental model.
  • You need K8s-ecosystem operators (Argo, Flux, Crossplane, Istio, Linkerd).
  • You want fine-grained network policies via Cilium / Calico.

When ECS wins:

  • AWS-only shop with no K8s expertise.
  • You want lower cognitive overhead.
  • Steady simple workloads where K8s features go unused.

Cost notes

  • Cluster fee: $73/month per EKS cluster. Don’t run 50 clusters; use namespaces.
  • Nodes: Karpenter with Spot + ARM saves 50-80% vs Reserved x86 on-demand.
  • Fargate: ~20% premium over EC2 equivalent; no node ops to amortize.
  • NAT Gateway is the surprise bill on EKS. Use VPC Endpoints for S3/ECR; consider IPv6 to skip NAT.

Interview angle

  • “EKS vs ECS — when each?” — EKS for K8s expertise / multi-cloud / rich ecosystem (Argo, Istio); ECS for AWS-native simplicity, no K8s control plane cost, simpler mental model. Both can run on Fargate.
  • “What’s IRSA?” — IAM Roles for Service Accounts. K8s ServiceAccount annotated with an IAM role ARN; AWS SDK in the pod assumes that role via OIDC. Per-pod IAM least-privilege, no long-lived keys in the cluster.
  • “Karpenter vs Cluster Autoscaler?” — Cluster Autoscaler scales node groups (ASGs); Karpenter provisions EC2 directly per Pending pod, picks instance type from allowed shapes, consolidates underutilized nodes automatically. Karpenter is the modern default on EKS.
  • “How do you handle secrets in EKS?” — External Secrets Operator + IRSA: ESO pulls from Secrets Manager / SSM as the operator’s IAM role; materializes k8s Secrets that pods consume. Or Secrets Store CSI driver — mount secrets as files directly without etcd storage.
  • “VPC CNI vs Cilium?” — VPC CNI: AWS default; pods are first-class VPC citizens with real subnet IPs. Cilium: eBPF-based; richer NetworkPolicies, host firewall, observability without sidecar. Pick Cilium for advanced network policy / observability needs.
  • “EKS Fargate trade-offs?” — no node ops (great for small / bursty workloads, CI), but ~20% premium and not all features supported (no DaemonSets, no privileged pods, limited storage). Mix with EC2 nodes for steady production.