backend / cloud aws / databases / README.md

Amazon DynamoDB

6 min read index source

Amazon DynamoDB

Fully-managed NoSQL key-value + document store. Single-digit millisecond latency at any scale. Different programming model from SQL — design starts from access patterns, not from entities.

Core model

  • Table has a primary key.
  • Primary key is either (partition key) alone or (partition key, sort key) — composite.
  • Partition key determines which physical partition stores the item; sort key orders items within that partition.
  • Items are schemaless JSON-like documents; only the primary key is required.
import boto3
ddb = boto3.resource("dynamodb")
orders = ddb.Table("orders")

orders.put_item(Item={
    "pk": "USER#42",
    "sk": "ORDER#2024-01-15T10:30:00Z#abc123",
    "total": 99.50,
    "items": [{"sku": "X1", "qty": 2}],
})

# Query: all orders for a user, sorted by time descending
resp = orders.query(
    KeyConditionExpression="pk = :pk AND sk BETWEEN :a AND :b",
    ExpressionAttributeValues={":pk": "USER#42", ":a": "ORDER#2024-01-01", ":b": "ORDER#2024-12-31"},
    ScanIndexForward=False,
)

Partition keys — the design that matters most

DynamoDB shards by hash(partition_key) → physical partition. All items with the same partition key live on the same partition, sorted by sort key, capped at ~10GB and 3000 RCU / 1000 WCU per partition.

This is the source of all the design trickiness.

Hot partition antipattern

pk = "ORDER"               # everything in one partition → throttled
pk = current_date          # today's writes all hit one partition
pk = user_id (single user) # one VIP user dominates throughput

Fix with high-cardinality keys + composite sort key

pk = USER#<user_id>        # distributes writes across users
sk = ORDER#<iso_timestamp>#<order_id>   # sort orders by time within user

For “global feed” use cases without a natural high-cardinality key, write-sharding:

pk = "FEED#<random_bucket_0_to_9>"
sk = "<timestamp>#<id>"

Reads fan out across N partitions in parallel.

Single-table design

DynamoDB best practice: one table with overloaded keys to handle many entity types and many access patterns. Look up by (pk, sk); GSIs for alternate access.

pk             | sk                  | data
USER#42        | PROFILE             | { name: ..., email: ... }
USER#42        | ORDER#2024-01-15... | { total: ..., items: ... }
USER#42        | ORDER#2024-01-20... | { total: ..., items: ... }
USER#42        | ADDR#home           | { street: ... }
ORDER#abc123   | METADATA            | { user_id: 42, status: ... }

One query pk = USER#42, sk BETWEEN ORDER# AND ORDER#~ returns all orders for user 42. Another pk = USER#42, sk = PROFILE returns the profile.

This breaks every RDBMS instinct. Use it; it’s why DynamoDB scales.

Secondary indexes

Global Secondary Index (GSI)

Different partition key + sort key, projected attributes. Stored as separate table; eventually consistent. Reads/writes consume separate capacity.

# GSI to look up orders by status, sorted by created_at
orders.query(
    IndexName="status-created_at-index",
    KeyConditionExpression="status = :s AND created_at > :t",
    ExpressionAttributeValues={":s": "PENDING", ":t": "2024-01-01"},
)

5 GSIs per table by default. Hot-partition rules apply to the GSI’s partition key.

Local Secondary Index (LSI)

Same partition key, different sort key. Strongly consistent reads. Must be created at table creation; can’t add later.

GSIs are far more common. Skip LSIs unless you specifically need strong consistency.

Capacity modes

On-demand

Pay per request. No capacity planning. Auto-scales instantly. Pricier per request but no provisioning.

Best for: spiky / unpredictable workloads, dev/staging, new applications.

Provisioned

You set RCUs (Read Capacity Units) and WCUs (Write Capacity Units). Auto-scaling adjusts based on utilization target.

  • 1 RCU = 1 strongly-consistent read of an item ≤ 4KB per second.
  • 1 WCU = 1 write of an item ≤ 1KB per second.

Best for: predictable workloads where you can right-size.

Switching modes is allowed but rate-limited (~once per day).

Streams

Change Data Capture — every write to the table emits a record to a stream with the old/new image.

# Stream record consumed by Lambda
{
  "eventName": "INSERT" | "MODIFY" | "REMOVE",
  "dynamodb": {
    "Keys": {...},
    "NewImage": {...},
    "OldImage": {...},
    "SequenceNumber": "...",
  }
}

Used for:

  • Triggering downstream processing (Lambda).
  • Replicating to ElasticSearch / OpenSearch for search.
  • Transactional outbox pattern (write to DynamoDB → stream → Kafka).

DynamoDB Streams + Lambda gives you reactive data processing without managing brokers.

TTL

Set an attribute to a Unix timestamp; DynamoDB deletes items at that time (~48h lag). Free, but lazy.

orders.put_item(Item={
    "pk": "SESSION#xyz",
    "sk": "TOKEN",
    "expires_at": int(time.time()) + 3600,  # 1 hour from now
})
# enable TTL on attribute "expires_at" once at table create

Useful for session tokens, idempotency keys, rate-limit windows.

Transactions

Up to 100 items across one or more tables, ACID:

client = boto3.client("dynamodb")
client.transact_write_items(TransactItems=[
    {"Update": {"TableName": "accounts", "Key": {"id": {"S": "A"}}, ...}},
    {"Update": {"TableName": "accounts", "Key": {"id": {"S": "B"}}, ...}},
    {"Put": {"TableName": "ledger", "Item": {...}}},
])

Costs 2x WCU per item. Conditional checks (ConditionExpression) provide optimistic concurrency.

Pagination

Queries return up to 1MB; the response includes LastEvaluatedKey if more data.

last = None
while True:
    kwargs = {"KeyConditionExpression": "pk = :p", "ExpressionAttributeValues": {":p": "USER#42"}}
    if last:
        kwargs["ExclusiveStartKey"] = last
    resp = orders.query(**kwargs)
    for item in resp["Items"]:
        process(item)
    last = resp.get("LastEvaluatedKey")
    if not last:
        break

Scan is the same but reads the entire table — avoid in production.

Common gotchas

  • Hot partitions. A single popular pk = everyone is on one shard = throttling. Use high-cardinality keys; write-shard if needed.
  • Scan in code paths. Reads the entire table; throttles your app. Lift it into a query or rethink the access pattern.
  • Item size limit 400KB. Big payloads → S3 + pointer.
  • Eventual consistency by default. Pass ConsistentRead=True for strongly consistent (costs 2x RCU).
  • Reserved attribute names (name, status, type, …) — use ExpressionAttributeNames to alias them.
  • DynamoDB Local for tests. It exists; behavior is close to but not identical to prod.

Common interview question: “Design a tweet feed”

# Users table
pk = USER#<user_id>, sk = PROFILE         { name, ... }
pk = USER#<user_id>, sk = FOLLOWS#<other> { ... }     # who I follow

# Tweets
pk = TWEET#<tweet_id>, sk = METADATA      { user_id, body, ts }

# Feed (fanout-on-write)
pk = FEED#<user_id>, sk = <ts>#<tweet_id> { tweet_id, author_id }

Tweet from user → fanout: write a FEED row for each follower (write-amplification, but read is one-query). For celebrities with millions of followers, hybrid: pre-fanout for normal users, fan-out-on-read (pull from followed users’ tweet partitions) for celebrities.

When NOT to use DynamoDB

  • You need joins, GROUP BY, ad-hoc analytics → use Postgres.
  • Access patterns aren’t known up front and change frequently → SQL is more flexible.
  • Items > 400KB common → wrong store.
  • You need server-side aggregations on big data → use Redshift / Athena.

Interview angle

  • “Partition key design — why does it matter?” — DynamoDB shards by hash(pk); all items with the same pk live in one partition, capped on throughput. Bad pk → hot partition → throttling. High-cardinality pks distribute load.
  • “What’s single-table design?” — one table for many entity types, with overloaded pk/sk to encode entity + relationship. Optimized for known access patterns. Trades the flexibility of an RDBMS schema for the scale and performance of DynamoDB.
  • “GSI vs LSI?” — GSI: different pk, eventually consistent, addable any time, separate capacity. LSI: same pk + different sk, strongly consistent, must create at table creation. GSI is the common one.
  • “On-demand vs provisioned capacity?” — on-demand: pay per request, no planning, auto-scales instantly. Provisioned: you set RCUs/WCUs, auto-scale on utilization. On-demand for spiky/unknown; provisioned + reserved for steady predictable.
  • “How would you do a transactional outbox with DynamoDB?” — write business item + outbox marker in one TransactWriteItems call. DynamoDB Stream → Lambda → publish to Kafka/SNS. Outbox marker has TTL so it’s auto-cleaned.
  • “What’s a hot partition and how do you avoid one?” — disproportionate traffic to one pk → that one partition saturates while others sit idle → throttling. Use high-cardinality keys; write-shard (“FEED#0” through “FEED#9”) for unavoidable single-entity workloads; read from all shards in parallel.