Vector Databases

A vector database stores high-dimensional embeddings (typically 384-1536 dimensions) and supports fast approximate nearest-neighbor (ANN) search. The infrastructure underneath RAG.

For RAG patterns see 03_rag_architecture_patterns.md. For embeddings see 02_what_are_embeddings.md.

What a vector DB does

# Index a document
embedding = model.encode("The quick brown fox jumps over the lazy dog")
vector_db.upsert(
    id="doc_42",
    embedding=embedding,        # 768-dim vector
    metadata={"source": "stories.pdf", "author": "Anonymous"},
)

# Query
query_emb = model.encode("animals in literature")
results = vector_db.search(query_emb, top_k=5)
# Returns top-5 most similar documents by cosine similarity

Three jobs:

1. Store vectors + metadata.
2. Index them for fast nearest-neighbor search.
3. Query by similarity.

Optional fourth: filter by metadata while searching (“find similar AND published after 2024”).

Why approximate search

Exact nearest-neighbor in high dimensions is O(N) per query — scan everything. For millions of vectors that’s seconds per query. Not viable.

ANN (Approximate Nearest Neighbor) algorithms trade accuracy for speed: get 95-99% of the truly-nearest vectors in milliseconds.

The exactness/speed knob is called recall (the % of true top-K that you actually find). Higher recall = slower.

Index algorithms

Most production systems use HNSW or HNSW+PQ. HNSW is the default in Pinecone, Qdrant, Weaviate, pgvector (HNSW since 0.5).

The vector DB landscape

For most teams: pgvector if you’re on Postgres and have moderate scale (<10M vectors); Pinecone / Weaviate / Qdrant if you need a dedicated vector engine; FAISS for prototypes or embedded use.

Why pgvector deserves consideration

CREATE EXTENSION vector;

CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,
    content TEXT,
    embedding VECTOR(1536),
    metadata JSONB
);

CREATE INDEX ON documents USING HNSW (embedding vector_cosine_ops);

-- Hybrid query: vector similarity AND metadata filter AND text match
SELECT id, content
FROM documents
WHERE metadata->>'category' = 'engineering'
  AND content ILIKE '%kubernetes%'
ORDER BY embedding <=> '[0.1, 0.2, ...]'
LIMIT 10;

You get vector search + SQL joins + transactional consistency + your existing ops knowledge. For scales up to ~10M vectors, no separate service. Past that, performance tuning gets harder.

Similarity metrics

The distance function used. Different metrics give different rankings:

Most embedding models (OpenAI text-embedding-3, sentence-transformers, Cohere) recommend cosine. Always check the model’s documentation — using the wrong metric tanks quality.

Filtering

Pure vector search returns “most similar by embedding.” But you often need:

Find similar documents that ALSO:
- belong to user X (multi-tenancy)
- were published in 2024
- have type "blog"

Pre-filtering: filter first, then search remaining vectors. Slow if the filter excludes most data (search becomes brute force on what’s left).

Post-filtering: search first, then filter results. Fast but may return fewer than K results.

Filtered HNSW (modern vector DBs): the index understands filters; navigation respects them. Best of both. Most production-grade vector DBs (Qdrant, Pinecone, Weaviate) support this.

# Qdrant example with filter
results = client.search(
    collection_name="documents",
    query_vector=embedding,
    query_filter={"must": [
        {"key": "user_id", "match": {"value": user_id}},
        {"key": "published_at", "range": {"gte": "2024-01-01"}},
    ]},
    limit=5,
)

Multi-tenancy

For SaaS RAG, each customer’s data must be isolated. Three patterns:

Pinecone has namespaces; Qdrant has partitions; pgvector relies on metadata + indexes.

The shared+filter approach is the easy default but security-critical: every query MUST include the tenant filter. Centralize in a wrapper to avoid forgetting.

Updating and deletion

Reindexing for model changes (switching from text-embedding-ada-002 to text-embedding-3-large) requires re-embedding everything. Plan for it; running both old and new models during transition is common.

Most vector DBs support upserts cleanly. Pure libraries (FAISS) typically don’t — you rebuild or maintain side-by-side indices.

Scale

Rough capacity per node (approximate, varies by config):

Latency targets: 10-100ms for typical RAG queries. Anything past 500ms hurts the UX.

Memory is the big constraint. HNSW lives in RAM; a 1M × 1536-dim float32 index ≈ 6 GB. PQ compression can reduce 4-8×.

Hybrid search

Many production systems combine dense (vector) and sparse (BM25/keyword) retrieval:

# Dense: semantic similarity
dense_results = vector_db.search(query_embedding, top_k=20)

# Sparse: lexical match
sparse_results = bm25_index.search(query_text, top_k=20)

# Fuse with Reciprocal Rank Fusion (RRF)
fused = rrf_merge(dense_results, sparse_results)

Dense catches paraphrases (“password reset” matches “how to change password”). Sparse catches exact terms (“error code XYZ” matches the exact code).

Weaviate, OpenSearch, Vespa, Elasticsearch, and others have hybrid built in. Otherwise you run two indices and merge in code.

Cost considerations

For small-to-medium scale: pgvector is dramatically cheaper. For large scale with ops bandwidth: managed.

Hidden costs: embedding API calls (OpenAI ~$0.10/M tokens; can be huge for large corpora), bandwidth for retrieving full documents, re-embedding when models update.

Common pitfalls

• Wrong distance metric: model recommends cosine; you configured L2. Quality tanks silently.
• No metadata filtering: every query returns everyone’s data; multi-tenancy breach.
• Naive top-K: returning K=5 when the question needs 50 → missing info; returning K=100 → context bloat.
• Stale embeddings: documents updated but embeddings not regenerated. Search returns the old text.
• Embedding model drift: upgrading models without re-indexing → mismatched query/document spaces; recall plummets.
• Vector size mismatch: model outputs 1536-dim; index expects 768. Silent failure or error.
• No deduplication: chunked documents produce many near-identical vectors; top-K is redundant.
• Hot-tenant problem: one tenant has 99% of vectors; queries for that tenant are slow; small tenants get diluted.

Production checklist

• Cosine vs L2 vs dot product matches the embedding model.
• Multi-tenancy strategy (filter / namespace / index per tenant).
• Reindexing plan when models change.
• Backup and disaster recovery.
• Index parameters tuned (HNSW M, ef_construction, ef_search).
• Query latency monitored (p50, p99).
• Recall measured on a held-out set.
• Hybrid search if needed (dense + sparse).
• Pre-warming / connection pooling for low cold-start.

Choosing a vector DB

There’s no universally best vector DB. The decision is about your scale, existing infrastructure, and budget.

Common interview confusions

• “Vector DBs are slow.” — they do millisecond ANN search on millions of vectors. Slower than primary-key lookup; much faster than scanning.
• “FAISS is a vector database.” — FAISS is a library — no server, no persistence, no concurrent access. Vector DBs are services built on top of libraries like FAISS, HNSWlib.
• “Cosine and dot product are different.” — for normalized embeddings (||v|| = 1), they’re equivalent. Most modern embedding models output normalized vectors.

Interview angle

• “What is a vector database?” — storage + index for high-dimensional embeddings supporting fast approximate nearest-neighbor search. Used for RAG (retrieving relevant chunks), recommendations, search, anomaly detection.
• “How does approximate nearest neighbor search work?” — algorithms like HNSW build a graph or tree structure; queries navigate it instead of scanning every vector. Trade accuracy (recall) for speed (latency). HNSW is the dominant algorithm.
• “pgvector vs Pinecone?” — pgvector if you’re on Postgres and have <10M vectors; you get vector search + SQL + transactions in one. Pinecone for large scale or zero-ops preference. Cost crossover is around ~$200/mo Pinecone vs pgvector.
• “How do you handle multi-tenant vector search?” — three patterns: index per tenant (best isolation, more overhead), shared index + metadata filter (simple, must apply filter), namespaces/partitions (middle ground). Most production systems use namespaces or filters.
• “What’s hybrid search?” — combine dense (vector) retrieval with sparse (BM25/keyword) retrieval; fuse with Reciprocal Rank Fusion. Dense catches paraphrases; sparse catches exact terms. Better recall than either alone.
• “When would you NOT use a vector DB?” — small corpora (<10k chunks, FAISS in-memory is fine), purely keyword-based search (BM25 alone), simple lookup by ID (just use Postgres). Vector DBs add complexity; only use when you need semantic similarity.
• “What happens if you switch embedding models?” — old embeddings live in a different space than new ones; cannot compare. Must re-embed everything. Typical migration runs both models in parallel during transition.
• “Cosine similarity vs L2 distance — when?” — cosine for text embeddings (most modern models are trained on cosine); L2 for image features or when magnitude matters. Check your embedding model’s recommended metric.