Amazon S3 (Simple Storage Service)

Object storage. The oldest AWS service (2006) and the most foundational — most other services touch S3. Eleven-nines durability, virtually unlimited scale, accessible from anywhere by URL.

The model

• Bucket — a named container, globally unique across AWS.
• Object — a file + metadata. Key is the “path”, but S3 is a flat namespace (no real directories).
• Region — buckets are regional; replicate across regions explicitly.

import boto3
s3 = boto3.client("s3")
s3.put_object(Bucket="my-app-uploads", Key="users/42/avatar.png", Body=image_bytes,
              ContentType="image/png")
obj = s3.get_object(Bucket="my-app-uploads", Key="users/42/avatar.png")
data = obj["Body"].read()

Storage classes

For most apps: Intelligent-Tiering on user uploads + Standard on hot operational data. Lifecycle rules transition older objects to Glacier.

Lifecycle policies

{
  "Rules": [{
    "ID": "transition-and-expire",
    "Status": "Enabled",
    "Filter": {"Prefix": "logs/"},
    "Transitions": [
      {"Days": 30,  "StorageClass": "STANDARD_IA"},
      {"Days": 90,  "StorageClass": "GLACIER"},
      {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
    ],
    "Expiration": {"Days": 2555}
  }]
}

After 30 days, transition to IA; after 90 to Glacier; expire (delete) at 7 years. Saves an enormous amount of money on log data.

Versioning

Per-bucket toggle. After enable, overwrites and deletes preserve old versions; you can list and restore.

s3 = boto3.client("s3")
versions = s3.list_object_versions(Bucket="orders-archive", Prefix="reports/2024-01")

Delete protection: a “delete” actually creates a delete marker; the object still exists at the previous version. MFA Delete requires MFA token to delete versions, for paranoid setups.

Side effect: storage grows. Pair with lifecycle rules to expire old versions:

{"NoncurrentVersionExpiration": {"NoncurrentDays": 30}}

Presigned URLs

Generate a URL with embedded auth + expiration; share with anyone:

url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "uploads", "Key": "report.pdf"},
    ExpiresIn=3600,  # 1 hour
)

The dominant pattern for direct browser uploads / downloads without proxying through your app:

# Direct upload: backend generates presigned PUT URL, client uploads directly
url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "uploads", "Key": f"users/{user_id}/avatar.png", "ContentType": "image/png"},
    ExpiresIn=900,
)
return {"upload_url": url}

The client PUTs directly to S3, your backend never sees the bytes. Huge for media-heavy apps.

For form-based uploads (with field restrictions): generate_presigned_post.

Encryption

• SSE-S3 — AWS-managed keys; free; default on new buckets.
• SSE-KMS — AWS KMS keys; per-API-call KMS cost; auditable in CloudTrail.
• SSE-C — customer-provided keys (rare).
• Client-side — encrypt before upload.

For most regulated workloads: SSE-KMS with a customer-managed CMK. The CMK key policy controls who can decrypt — separates “can access S3” from “can read the data”.

Access control

Three layers, evaluated together:

1. Bucket policy — JSON IAM-style policy on the bucket. Allow/deny by principal, action, resource, condition.
2. IAM policies — on users/roles. Standard IAM.
3. ACLs (legacy) — disabled on new buckets; don’t use.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowAccountReadOnly",
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::123456789012:role/app"},
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]
  }]
}

Block Public Access is the safety net. Turn on at the bucket and account level unless you’re consciously running a public bucket (static site hosting).

S3 + CloudFront for static sites

Bucket holds the static files; CloudFront fronts them with TLS, caching, custom domain, WAF. Origin Access Control (OAC) restricts the bucket to only CloudFront — no direct internet access.

client → CloudFront → S3 (private, OAC)

Cheap, fast, scalable. The pattern behind 99% of “static SPAs on AWS”.

S3 Events

S3 emits events on object creation / deletion. Targets:

• Lambda — process new uploads.
• SQS — buffer for batch processing.
• SNS — fan-out.
• EventBridge — richer event routing.

# Lambda triggered on s3:ObjectCreated:*
def handler(event, context):
    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = record["s3"]["object"]["key"]
        process(bucket, key)

The “image uploaded → resize” pipeline is the canonical example.

Object Lambda

Transform on the fly between S3 and the client — masking PII, resizing images, redacting columns from CSVs — without storing the variant.

# Object Lambda function
def handler(event, context):
    s3 = boto3.client("s3")
    original = requests.get(event["getObjectContext"]["inputS3Url"]).content
    transformed = mask_pii(original)
    s3.write_get_object_response(
        RequestRoute=event["getObjectContext"]["outputRoute"],
        RequestToken=event["getObjectContext"]["outputToken"],
        Body=transformed,
    )

Multipart upload

For objects > 5GB (mandatory) or large objects on flaky networks (recommended). Boto3 handles it transparently via upload_file:

s3.upload_file("big_video.mp4", "uploads", "videos/abc.mp4")
# Internally: multipart if > threshold

Failed multipart uploads leave orphan parts costing storage. Lifecycle rule to clean:

{"AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}}

Strong consistency

S3 has read-after-write consistency for all operations since 2020. PUT then GET sees the new object. No more eventual-consistency surprises.

But: list-after-write consistency is also strong now. ListObjects immediately after a PutObject includes the new key.

Common gotchas

• ListObjects paginates at 1000. Use list_objects_v2 with ContinuationToken or paginator.
• Path-style vs virtual-hosted-style URLs. Path-style is being deprecated; use bucket.s3.region.amazonaws.com URLs.
• Default bucket-owner-full-control. Cross-account writers need to set ACL or you can’t read their uploads.
• CORS for browser uploads. Set CORS on the bucket; otherwise direct uploads via presigned URLs blow up.
• Naming a bucket the same as another in the world. Bucket names are globally unique; my-app is taken since 2007.
• HEAD before GET to check existence. head_object raises 404 on missing; cleaner than get_object + catch.
• Big bills from S3 Standard on cold data. Always set lifecycle rules; without them, logs from 5 years ago still cost full price.

S3 Select / Athena

Query S3 data without downloading:

• S3 Select — SQL on a single object (CSV, JSON, Parquet). Cheap if you grep one file.
• Athena — serverless SQL across many objects, partitioned. Pay per TB scanned.

For analytics on S3 data lakes: Athena (or Redshift Spectrum, or Trino).

Common interview pattern: “upload + process”

1. Backend generates presigned PUT URL → client.
2. Client uploads directly to S3.
3. S3 emits ObjectCreated event → SQS.
4. Lambda / Fargate worker reads SQS, processes the object.
5. Worker writes result back to S3 / DB.
6. (Optional) Notify user via WebSocket / push.

Critical: idempotent processing, since SQS delivery is at-least-once.

Interview angle

• “How do you handle large file uploads from a browser without proxying through your backend?” — backend generates a presigned PUT URL (with Content-Type, Content-Length constraints, short expiry); client uploads directly to S3. Or generate_presigned_post for form-based uploads. Set bucket CORS.
• “What storage class for log files?” — Standard for the first 30 days (probably grepped during incidents), transition to Standard-IA at 30d, Glacier at 90d, expire at retention horizon. Lifecycle policy does this automatically.
• “How do you secure a private S3 bucket fronted by CloudFront?” — Origin Access Control (OAC). Bucket policy allows the CloudFront distribution; bucket has Block Public Access on. Clients can only reach the content via CloudFront (TLS + caching + WAF).
• “What’s the consistency model?” — strong read-after-write since Dec 2020. PUT then immediate GET sees the new object. List operations are also strongly consistent now.
• “How do you trigger work on a new upload?” — S3 event notifications → SQS/Lambda/SNS/EventBridge. Pattern: S3 → SQS → worker (idempotent). Lambda direct works for low-volume; SQS gives buffering and retry.
• “How do you handle PII masking in objects?” — Object Lambda. Transforms responses on the GET path without storing variants. Or, for batch: separate “masked” bucket populated by a pipeline.