What is Embeddings?

Definition

Embeddings are dense vector representations of data (text, images, etc.) that capture semantic meaning in a continuous vector space. They convert discrete objects (words, sentences, documents) into numerical vectors that can be used for machine learning tasks. Similar objects have similar embeddings (close in vector space).

Key Concepts

Core Principles

Vector Representation: Objects are mapped to fixed-size numerical vectors
Semantic Similarity: Similar meanings → similar vectors (close in space)
Dense Vectors: High-dimensional vectors (typically 100-1536 dimensions)
Learned Representations: Embeddings are learned from data
Transfer Learning: Pre-trained embeddings can be used for various tasks

Why Embeddings Matter

Semantic Understanding: Captures meaning, not just syntax
Mathematical Operations: Can compute similarity, find nearest neighbors
Machine Learning Input: Provides numerical representation for ML models
Efficiency: Compact representation of complex information

Types of Embeddings

1. Word Embeddings

Represent individual words as vectors.

Examples:

Word2Vec
GloVe
FastText

from gensim.models import Word2Vec

# Train word embeddings
sentences = [
    ["machine", "learning", "is", "great"],
    ["deep", "learning", "uses", "neural", "networks"],
    ["natural", "language", "processing", "is", "fascinating"]
]

model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)

# Get word embedding
machine_vector = model.wv["machine"]
learning_vector = model.wv["learning"]

# Find similar words
similar = model.wv.most_similar("learning", topn=5)
# [('machine', 0.85), ('neural', 0.82), ...]

2. Sentence Embeddings

Represent entire sentences as vectors.

Examples:

Sentence-BERT (SBERT)
Universal Sentence Encoder
InferSent

from sentence_transformers import SentenceTransformer

# Load pre-trained model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Encode sentences
sentences = [
    "Machine learning is a subset of AI",
    "Deep learning uses neural networks",
    "The weather is nice today"
]

embeddings = model.encode(sentences)

# Calculate similarity
from sklearn.metrics.pairwise import cosine_similarity

similarity = cosine_similarity([embeddings[0]], [embeddings[1]])
# High similarity between first two sentences

3. Document Embeddings

Represent entire documents as vectors.

from langchain.embeddings import OpenAIEmbeddings

embeddings_model = OpenAIEmbeddings()

documents = [
    "This is a long document about machine learning...",
    "Another document about natural language processing...",
    "A document about computer vision..."
]

# Generate document embeddings
doc_embeddings = embeddings_model.embed_documents(documents)

# Each document is now a 1536-dimensional vector
print(len(doc_embeddings[0]))  # 1536

4. Contextual Embeddings

Word embeddings that change based on context.

Examples:

BERT
ELMo
GPT embeddings

from transformers import AutoTokenizer, AutoModel
import torch

# Load BERT model
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')

# Same word, different contexts
sentence1 = "I deposited money in the bank"
sentence2 = "I sat by the river bank"

# Tokenize
inputs1 = tokenizer(sentence1, return_tensors='pt')
inputs2 = tokenizer(sentence2, return_tensors='pt')

# Get embeddings
with torch.no_grad():
    outputs1 = model(**inputs1)
    outputs2 = model(**inputs2)

# "bank" has different embeddings in different contexts
bank_embedding1 = outputs1.last_hidden_state[0][2]  # "bank" in sentence1
bank_embedding2 = outputs2.last_hidden_state[0][6]  # "bank" in sentence2

# These embeddings are different!

How Embeddings Work

Basic Process

Input: Text/data to embed
Tokenization: Break into tokens (words, subwords)
Encoding: Convert tokens to numerical IDs
Embedding Lookup: Map IDs to embedding vectors
Aggregation: Combine token embeddings (for sentences/documents)
Output: Final embedding vector

Mathematical Representation

Text: "machine learning"
    ↓ Tokenization
Tokens: ["machine", "learning"]
    ↓ Encoding
IDs: [1234, 5678]
    ↓ Embedding Lookup
Vectors: [[0.1, 0.2, ...], [0.3, 0.4, ...]]
    ↓ Aggregation (mean, sum, etc.)
Final Embedding: [0.2, 0.3, ...]

Common Embedding Models

1. Word2Vec

from gensim.models import Word2Vec

# Training data
sentences = [
    ["king", "queen", "royalty"],
    ["man", "woman", "person"],
    ["paris", "france", "europe"]
]

# Train model
model = Word2Vec(
    sentences,
    vector_size=100,      # Embedding dimension
    window=5,             # Context window
    min_count=1,          # Minimum word frequency
    sg=0                  # 0=CBOW, 1=Skip-gram
)

# Use embeddings
king_vector = model.wv["king"]
queen_vector = model.wv["queen"]

# Famous example: king - man + woman ≈ queen
result = model.wv.most_similar(
    positive=["king", "woman"],
    negative=["man"],
    topn=1
)
# [('queen', 0.85)]

2. Sentence-BERT (SBERT)

from sentence_transformers import SentenceTransformer

# Load pre-trained model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Encode sentences
sentences = [
    "The cat sits on the mat",
    "A feline is on the rug",
    "I love programming"
]

embeddings = model.encode(sentences)

# Calculate pairwise similarities
from sklearn.metrics.pairwise import cosine_similarity

similarity_matrix = cosine_similarity(embeddings)
# First two sentences have high similarity
print(similarity_matrix[0][1])  # ~0.75

3. OpenAI Embeddings

from openai import OpenAI

client = OpenAI(api_key="your-api-key")

# Generate embeddings
text = "Machine learning is fascinating"
response = client.embeddings.create(
    model="text-embedding-ada-002",
    input=text
)

embedding = response.data[0].embedding
print(len(embedding))  # 1536 dimensions

4. BERT Embeddings

from transformers import AutoTokenizer, AutoModel
import torch

# Load BERT
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')

# Encode text
text = "Hello, how are you?"
inputs = tokenizer(text, return_tensors='pt', padding=True, truncation=True)

# Get embeddings
with torch.no_grad():
    outputs = model(**inputs)
    
# Use [CLS] token embedding for sentence representation
sentence_embedding = outputs.last_hidden_state[0][0]  # [CLS] token

# Or average all token embeddings
token_embeddings = outputs.last_hidden_state[0]
sentence_embedding = torch.mean(token_embeddings, dim=0)

Use Cases

1. Semantic Search

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# Knowledge base
documents = [
    "Python is a programming language",
    "Machine learning uses algorithms",
    "The weather is sunny today"
]

# Create embeddings for all documents
doc_embeddings = model.encode(documents)

# User query
query = "What is Python?"
query_embedding = model.encode([query])[0]

# Find most similar document
similarities = np.dot(doc_embeddings, query_embedding)
most_similar_idx = np.argmax(similarities)

print(f"Most relevant: {documents[most_similar_idx]}")

2. Text Classification

from sklearn.linear_model import LogisticRegression
from sentence_transformers import SentenceTransformer

# Prepare data
train_texts = ["I love this", "This is terrible", ...]
train_labels = [1, 0, ...]  # positive/negative

# Generate embeddings
model = SentenceTransformer('all-MiniLM-L6-v2')
train_embeddings = model.encode(train_texts)

# Train classifier
classifier = LogisticRegression()
classifier.fit(train_embeddings, train_labels)

# Predict
test_text = "This is amazing"
test_embedding = model.encode([test_text])
prediction = classifier.predict(test_embedding)

3. Clustering

from sklearn.cluster import KMeans
from sentence_transformers import SentenceTransformer

# Documents to cluster
documents = [
    "Machine learning algorithms",
    "Deep learning neural networks",
    "The weather forecast",
    "Today's temperature"
]

# Generate embeddings
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(documents)

# Cluster
kmeans = KMeans(n_clusters=2)
clusters = kmeans.fit_predict(embeddings)

# Results
for doc, cluster in zip(documents, clusters):
    print(f"Cluster {cluster}: {doc}")

4. Recommendation Systems

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# User's liked items
user_likes = [
    "Action movies with explosions",
    "Sci-fi space adventures"
]

# Available items
items = [
    "Action thriller with car chases",
    "Romantic comedy",
    "Space opera movie",
    "Documentary about nature"
]

# Embed user preferences
user_embedding = np.mean(model.encode(user_likes), axis=0)

# Embed items
item_embeddings = model.encode(items)

# Find similar items
similarities = np.dot(item_embeddings, user_embedding)
recommended_idx = np.argmax(similarities)

print(f"Recommended: {items[recommended_idx]}")

5. Anomaly Detection

from sentence_transformers import SentenceTransformer
from sklearn.ensemble import IsolationForest
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')

# Normal documents
normal_docs = [
    "Customer purchased product",
    "User logged into account",
    "Payment processed successfully"
]

# Generate embeddings
embeddings = model.encode(normal_docs)

# Train anomaly detector
detector = IsolationForest(contamination=0.1)
detector.fit(embeddings)

# Check new document
new_doc = "System error: unauthorized access"
new_embedding = model.encode([new_doc])
is_anomaly = detector.predict(new_embedding)

if is_anomaly[0] == -1:
    print("Anomaly detected!")

Embedding Dimensions

Common Dimensions

Word2Vec: 100-300 dimensions
GloVe: 50-300 dimensions
BERT: 768 dimensions (base), 1024 (large)
OpenAI Ada: 1536 dimensions
Sentence-BERT: 384-768 dimensions

Choosing Dimensions

Lower dimensions (50-100): Faster, less memory, may lose nuance
Medium dimensions (300-512): Good balance
Higher dimensions (768-1536): More expressive, slower, more memory

Similarity Metrics

1. Cosine Similarity

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

def cosine_similarity_custom(vec1, vec2):
    dot_product = np.dot(vec1, vec2)
    norm1 = np.linalg.norm(vec1)
    norm2 = np.linalg.norm(vec2)
    return dot_product / (norm1 * norm2)

# Example
vec1 = np.array([1, 2, 3])
vec2 = np.array([2, 4, 6])
similarity = cosine_similarity_custom(vec1, vec2)
print(similarity)  # 1.0 (perfectly similar direction)

2. Euclidean Distance

import numpy as np

def euclidean_distance(vec1, vec2):
    return np.linalg.norm(vec1 - vec2)

# Example
vec1 = np.array([1, 2, 3])
vec2 = np.array([2, 4, 6])
distance = euclidean_distance(vec1, vec2)
print(distance)  # 3.74 (lower = more similar)

3. Dot Product

import numpy as np

def dot_product_similarity(vec1, vec2):
    return np.dot(vec1, vec2)

# Example
vec1 = np.array([1, 2, 3])
vec2 = np.array([2, 4, 6])
similarity = dot_product_similarity(vec1, vec2)
print(similarity)  # 28 (higher = more similar, but not normalized)

Common Interview Questions and Answers

Q1: What are embeddings and why are they useful?

Embeddings are dense vector representations that capture semantic meaning. They’re useful because:

Semantic Understanding: Capture meaning, not just syntax
Mathematical Operations: Can compute similarity, perform arithmetic
ML Input: Provide numerical representation for machine learning
Efficiency: Compact representation of complex information
Transfer Learning: Pre-trained embeddings work across tasks

Q2: What’s the difference between word embeddings and sentence embeddings?

Aspect	Word Embeddings	Sentence Embeddings
Input	Single words	Entire sentences
Examples	Word2Vec, GloVe	Sentence-BERT, Universal Sentence Encoder
Use Case	Word-level tasks	Sentence-level tasks
Context	May not capture context	Captures sentence context

Word Embeddings: “bank” has same embedding regardless of context Sentence Embeddings: “bank” embedding changes based on sentence context

Q3: How do you create embeddings for a custom domain?

Fine-tune Pre-trained Model:

from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

# Load base model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Prepare domain-specific data
train_examples = [
    InputExample(texts=["medical term 1", "medical term 2"], label=0.9),
    InputExample(texts=["medical term 3", "unrelated term"], label=0.1)
]

# Fine-tune
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)
train_loss = losses.CosineSimilarityLoss(model)

model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=10
)

Train from Scratch (if you have large domain corpus):

from gensim.models import Word2Vec

# Domain-specific corpus
domain_sentences = load_domain_corpus()

# Train
model = Word2Vec(domain_sentences, vector_size=300, window=5)

Q4: What is the curse of dimensionality in embeddings?

As dimensions increase:

Sparsity: Most vectors become far apart
Distance metrics become less meaningful: All distances become similar
Computational cost: Storage and computation increase
Overfitting risk: Model may memorize rather than generalize

Solutions:

Use dimensionality reduction (PCA, t-SNE)
Choose appropriate embedding dimensions
Use regularization

Q5: How do you handle out-of-vocabulary (OOV) words?

Subword Tokenization (BPE, WordPiece):

# BERT uses WordPiece
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokens = tokenizer.tokenize("unhappiness")
# ['un', '##happy', '##ness'] - can handle unknown words

Character-level Embeddings:

# FastText uses character n-grams
from gensim.models import FastText
model = FastText(sentences, vector_size=100)
# Can generate embedding for any word

Fallback Strategies:
- Use average of known words
- Use special UNK token embedding
- Use character-level or subword embeddings

Q6: What’s the difference between static and contextual embeddings?

Static Embeddings (Word2Vec, GloVe):

Same word always has same embedding
“bank” (financial) and “bank” (river) have same vector
Fast, efficient
Don’t capture context

Contextual Embeddings (BERT, ELMo):

Same word has different embeddings based on context
“bank” in different sentences has different vectors
More expressive, captures context
Slower, requires more computation

Q7: How do you choose the right embedding model?

Consider:

Task Type:
- Word-level: Word2Vec, GloVe
- Sentence-level: Sentence-BERT, Universal Sentence Encoder
- Contextual: BERT, RoBERTa
Language:
- English: Many options
- Other languages: Check multilingual models (mBERT, XLM-R)
Domain:
- General: Use pre-trained models
- Specialized: Fine-tune or train domain-specific
Performance Requirements:
- Fast inference: Smaller models (DistilBERT, MiniLM)
- Best quality: Larger models (BERT-large, GPT)
Resources:
- Limited: Use smaller models or APIs (OpenAI)
- Available: Can train/fine-tune larger models

Q8: How do embeddings capture semantic relationships?

Through training on large text corpora, embeddings learn:

Distributional Hypothesis: Words in similar contexts have similar meanings
Vector Arithmetic: Relationships can be captured mathematically
```
# Famous example
king - man + woman ≈ queen
```
Clustering: Similar concepts cluster together in vector space
Distance: Semantic similarity ≈ vector distance

Q9: What is embedding normalization and when to use it?

Normalization scales embeddings to unit length.

import numpy as np

def normalize_embedding(embedding):
    norm = np.linalg.norm(embedding)
    return embedding / norm if norm > 0 else embedding

# Benefits:
# 1. Cosine similarity = dot product (faster)
# 2. All embeddings on same scale
# 3. Better for some algorithms (k-means, etc.)

When to use:

When using cosine similarity (makes it = dot product)
For clustering algorithms
When embeddings have different scales
For better numerical stability

Q10: How do you evaluate embedding quality?

Intrinsic Evaluation:
- Word similarity tasks (SimLex, WordSim)
- Analogy tasks (king - man + woman = queen)
- Clustering quality
Extrinsic Evaluation:
- Downstream task performance (classification, NER)
- Retrieval quality (precision, recall)
- Task-specific metrics
Qualitative Analysis:
- Nearest neighbors inspection
- Visualization (t-SNE, UMAP)
- Domain expert review

def evaluate_embeddings(model, test_pairs):
    """Evaluate on word similarity task"""
    similarities = []
    for word1, word2, human_score in test_pairs:
        emb1 = model.encode([word1])[0]
        emb2 = model.encode([word2])[0]
        pred_similarity = cosine_similarity([emb1], [emb2])[0][0]
        similarities.append((human_score, pred_similarity))
    
    # Calculate correlation
    correlation = np.corrcoef(
        [s[0] for s in similarities],
        [s[1] for s in similarities]
    )[0][1]
    
    return correlation

Best Practices

Choose Right Model: Match model to task (word/sentence/document)
Normalize When Needed: For cosine similarity, normalize embeddings
Handle OOV: Use subword tokenization or character-level
Fine-tune for Domain: Adapt pre-trained models to your domain
Cache Embeddings: Store computed embeddings for reuse
Batch Processing: Process multiple texts together for efficiency
Monitor Quality: Regularly evaluate embedding quality
Version Control: Track which embedding model/version you use
Dimensionality: Choose appropriate dimensions (not always more = better)
Update Regularly: Keep embeddings current with new data

Summary

Embeddings are:

Dense vector representations of text/data
Capture semantic meaning in numerical form
Enable similarity computation and ML tasks
Available in various types: word, sentence, document, contextual
Powerful tool for NLP, search, recommendation, and more

Key takeaways:

Embeddings convert discrete objects to continuous vectors
Similar objects have similar embeddings (close in vector space)
Pre-trained embeddings are powerful and widely available
Choose embedding type based on your task
Evaluate embeddings on your specific use case

Interview angle

“What is an embedding?” - a dense vector positioning text in a space where geometric closeness approximates semantic similarity. It’s what makes “find similar meaning” a nearest-neighbour lookup.
“Why normalise before comparing?” - raw dot product mixes direction (semantics) with magnitude, which often reflects length rather than meaning. Unit-normalising makes dot product exactly cosine similarity. See ../00_math_foundations/01_linear_algebra.md.
“How do you choose an embedding model?” - by your own retrieval eval, not a leaderboard. Then dimension (storage and latency cost), max input length against your chunk size, multilingual need, and whether it can run in your data boundary.
“What breaks when you change embedding model?” - everything already indexed. Vectors from different models are not comparable, so a model change means a full re-embed and re-index. Version the index and plan the migration.