MongoDB — Pitfalls and Performance

Common production traps and the senior-level fixes.

The 16 MB document limit

Documents are capped at 16 MB. Hard limit. Causes:

• Unbounded array growth — comments embedded into a blog post, infinite events array on a user.
• Large binary blobs — base64-encoded PDFs in a field.
• Aggregating into a single doc — $push everything into one array via $group.

Symptoms:

• BSONObjectTooLarge error on insert/update.
• Slow reads of bloated documents.
• Working set blowup — large docs evict smaller docs from RAM.

Fix patterns

For unbounded arrays: the bucket pattern.

// Bad — unbounded comments embedded
{ "_id": "post1", "comments": [c1, c2, ..., c10000] }

// Bucket pattern — N comments per bucket doc
{ "_id": ObjectId(), "post_id": "post1", "bucket": 0, "comments": [c1..c100] }
{ "_id": ObjectId(), "post_id": "post1", "bucket": 1, "comments": [c101..c200] }

Or just reference: comments in a separate collection.

For large blobs: GridFS splits files into 255 KB chunks across two collections (files.files, files.chunks). For uploads, use S3 / object storage and reference by URL — MongoDB is rarely the right place for blob storage.

Unbounded array growth (broader issue)

Even under 16 MB, growing arrays hurt:

• Each update to a doc with a large array rewrites the whole doc.
• Index entries grow (multikey indexes per array element).
• Memory pressure.

Cap arrays explicitly:

db.users.update_one(
    {"_id": user_id},
    {
        "$push": {"recent_events": {
            "$each": [new_event],
            "$slice": -100,         # keep only the last 100
        }}
    }
)

$slice: -N keeps only the last N elements. Bounded.

Working set vs RAM

MongoDB caches indexes + recently-accessed documents in WiredTiger’s cache (default ~50% of RAM). If your “working set” (data you actually query) exceeds cache, every read pages from disk → 100× slowdown.

Diagnose:

• db.serverStatus().wiredTiger.cache — bytes in cache, evictions per second.
• Slow queries on small result sets often mean cache miss.

Fixes:

• Reduce working set: archive old data, smaller documents, tighter projections.
• Add RAM or shard.
• Better indexes that touch fewer pages.

Slow queries

db.orders.find({ user_id: 7 }).explain("executionStats")

Look for:

• winningPlan.stage: "COLLSCAN" → no index. Add one.
• executionStats.totalDocsExamined / nReturned >> 1 → wrong index or inefficient predicate.
• executionStats.executionTimeMillis > 50 → investigate.

Profile slow operations:

db.setProfilingLevel(1, { slowms: 50 })   // log queries slower than 50ms
db.system.profile.find().sort({ ts: -1 }).limit(20)

Missing index on shard key (sharded clusters)

On a sharded collection, any query NOT including the shard key in the filter scatters to all shards. Symptoms:

• Latency rises as you add shards (more network hops, no parallelism).
• All shards report load on the same query.

Fix: include the shard key in queries. If you can’t, the shard key choice was wrong.

Atomic update gotchas

// Update only if no concurrent writer
db.accounts.update_one(
    {"_id": "alice", "version": current_version},
    {"$set": {"balance": new_balance}, "$inc": {"version": 1}}
)

Atomic at the single-doc level. But if result.matched_count == 0 on a unique-key update, you may have a race — check and retry.

Index pitfalls

• Too many indexes — 5-10% write cost per index. Each insert / update touches every relevant index. Audit and remove unused indexes:

  db.orders.aggregate([{$indexStats: {}}])

  Look for indexes with accesses.ops = 0 over the last week.

• Index size > RAM — index pages thrash from disk. Equally bad as missing index.

• Multi-field range without compound index — find({ a: {$gt: 5}, b: {$gt: 10} }) with separate indexes does intersection at best. Compound is needed for efficient range-on-multiple.

• $regex without anchor — /foo/ scans every doc; /^foo/ uses index prefix. The anchor matters.

Memory leak from huge result sets

for doc in db.orders.find({...}):   # cursor, OK — streams
    process(doc)

list(db.orders.find({...}))         # eager — loads everything into RAM

Cursors stream; list() / [doc for doc in ...] loads all results into memory. For large result sets, iterate the cursor.

For Motor (async):

async for doc in db.orders.find({...}):     # streams
    process(doc)

Schema drift

MongoDB doesn’t enforce schema by default. Production reality:

• Fields rename without migration → old code expects created_at, new docs have createdAt.
• Types drift → total is sometimes int, sometimes string, sometimes Decimal128.
• Optional fields proliferate.

Mitigations:

• Schema validation at the collection level (3.6+):

  db.createCollection("orders", {
      validator: {
          $jsonSchema: {
              required: ["user_id", "total"],
              properties: {
                  user_id: { bsonType: "objectId" },
                  total:   { bsonType: "decimal", minimum: 0 }
              }
          }
      }
  })

• Application-level schemas — Pydantic models for serialization/deserialization.
• Versioned documents — schema_version: 2 field; migration scripts to upgrade.

Read preference + write concern mismatches

client = MongoClient(..., w="majority", read_preference="secondaryPreferred")
db.users.insert_one({"_id": "alice"})
user = db.users.find_one({"_id": "alice"})    # might MISS — secondary lag

Solutions:

• Use causal-consistency session for read-your-write.
• Or force primary read preference for that operation.
• Or readConcern: "majority" to bound staleness.

See 04_transactions_concerns.md.

Counting without countDocuments

db.orders.count_documents({})              # exact, scans
db.orders.estimated_document_count()       # fast, from metadata, may be stale

For dashboards: estimated is usually fine. For pagination (“show 1 of 234 pages”): exact, but consider not showing total at all (use “next page” pattern).

Slow $lookup

$lookup (join) scans the foreign collection per input doc unless the foreign side is indexed on foreignField. Always index the join key on the foreign side.

For large joins, the pipeline form of $lookup with let + pipeline lets you pre-filter and project the foreign side, reducing work.

Connection pool exhaustion

client = MongoClient(..., maxPoolSize=100)

Default pool: 100. At high concurrency (FastAPI workers × many concurrent requests), can run out. Symptoms: requests block on acquire, p99 latency rises.

Fix: raise pool, or scale clients horizontally. Don’t create a MongoClient per request — instantiate once at startup.

Bulk operations

operations = [
    UpdateOne({"_id": 1}, {"$inc": {"count": 1}}),
    UpdateOne({"_id": 2}, {"$inc": {"count": 1}}),
    InsertOne({"_id": 3, "name": "x"}),
]
db.collection.bulk_write(operations, ordered=False)

Batches operations. ordered=False — failures don’t stop subsequent ops; parallelizable. Much faster than N round-trips for bulk inserts / updates.

Transactions are slow (recap)

• Use single-doc atomic operations ($set, $inc, $push) whenever possible.
• Multi-doc transactions add latency + memory + WriteConflict retry overhead.
• Sharded-cluster transactions are especially expensive.

If your workload routinely uses multi-doc transactions, consider whether the data model can be restructured to a single doc.

Monitoring essentials

Interview angle

• “What’s the 16 MB document limit and how do you work around it?” — hard cap per doc. Solutions: bucket pattern (split into multiple docs), reference (separate collection), or GridFS / S3 for true binary blobs. Don’t accumulate unbounded arrays.
• “How do you prevent unbounded array growth?” — $push with $slice: -N to keep only the last N elements. Or move the array to a separate collection (referenced) so the parent doesn’t grow.
• “Working set exceeds RAM — symptoms and fixes?” — every read pages from disk, ~100× slowdown. Fixes: archive cold data, tighter projections, add RAM, better indexes that touch fewer pages, or shard.
• “Why is your $lookup slow?” — no index on foreignField in the joined collection. Always index the join key. Pipeline-form $lookup with pre-filtering reduces work further.
• “You see BSONObjectTooLarge on update. What’s the cause?” — document with array growth or large embedded field pushed past 16 MB. Restructure with bucket pattern or reference.
• “Inserted a doc; read on a secondary doesn’t show it. Why?” — secondary lag + secondaryPreferred read preference. Either use causal consistency in a session, force primary read for that op, or readConcern: "majority" for stronger guarantees.
• “How do you find unused indexes?” — db.coll.aggregate([{$indexStats: {}}]). Indexes with low accesses.ops over a window are candidates for removal. Each removed index gives back write performance.