Dimensionality reduction

Fewer dimensions means less noise, less compute, and plots you can actually look at. The interview trap is confusing techniques meant for modelling with techniques meant only for visualisation.

Why reduce

• Curse of dimensionality — distances concentrate, so anything distance-based degrades. See 07_knn_naive_bayes.md.
• Compute and storage — 1536-dimensional embeddings times ten million documents is real money.
• Noise removal — low-variance directions are often measurement noise.
• Visualisation — humans read 2D.
• Collinearity — decorrelate before a linear model.

PCA

Find the orthogonal directions of greatest variance, project onto the top k.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95))  # keep 95% of variance
X_reduced = pipe.fit_transform(X)
print(pipe[-1].n_components_)

Mechanically it’s the eigendecomposition of the covariance matrix, or equivalently the SVD of the centred data. See ../00_math_foundations/01_linear_algebra.md.

Centring is required; scaling usually is. PCA maximises variance, so a feature measured in thousands dominates one measured in units purely by unit choice. Without scaling you’re doing PCA on your unit system.

Choosing n_components:

import numpy as np
pca = PCA().fit(X_scaled)
cumulative = np.cumsum(pca.explained_variance_ratio_)

Pass a float (0.95) to select by retained variance, an int for a fixed count, or look for the elbow in the scree plot.

Key properties:

• Linear. It can only find linear structure. Data on a curved manifold won’t unroll.
• Components are uninterpretable in general — each is a weighted mix of all original features. “PC1 is 0.3×income + 0.2×age − 0.4×tenure” is rarely a meaningful concept.
• Invertible, approximately: pca.inverse_transform reconstructs, and reconstruction error is a decent anomaly detector.
• Fit on training data only, then transform validation and test. Fitting on everything leaks.

Variants: TruncatedSVD for sparse matrices (it skips centring, which would destroy sparsity — the standard choice for TF-IDF), IncrementalPCA for out-of-core data.

t-SNE and UMAP: visualisation only

Both are non-linear and preserve local neighbourhood structure.

from sklearn.manifold import TSNE
import umap

X_tsne = TSNE(n_components=2, perplexity=30, init="pca").fit_transform(X)
X_umap = umap.UMAP(n_neighbors=15, min_dist=0.1).fit_transform(X)

UMAP is the default in 2026 — faster, keeps more global structure, and can embed new points without refitting.

The rules people break

Do not feed t-SNE/UMAP output into a classifier as features. They’re non-linear, non-deterministic, and (for t-SNE) can’t map new data. Use PCA if you want features. Use t-SNE/UMAP if you want a picture.

Cluster distances in a t-SNE plot are not meaningful. t-SNE preserves neighbourhoods, not distances. Two clusters appearing far apart may not be far apart. Cluster sizes in the plot are meaningless too.

The parameters change the picture qualitatively. Low perplexity fragments data into many small clumps; high perplexity merges them. Always look at several settings before believing a structure — a single t-SNE plot is not evidence.

Which to use

LDA — the supervised one

Linear Discriminant Analysis maximises between-class separation rather than total variance, so it uses labels.

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
X_lda = LinearDiscriminantAnalysis(n_components=2).fit_transform(X, y)

Limited to at most n_classes - 1 components. When the goal is class separability rather than variance, it beats PCA — a nice contrast to have ready.

(Not to be confused with Latent Dirichlet Allocation, the topic model. Same acronym, unrelated.)

Compressing embeddings in practice

Directly relevant to RAG cost, and a good place to sound current:

• Matryoshka embeddings — models trained so that truncating the vector to its first k dimensions still works. Cutting 1536 to 512 costs little quality and a third of the storage. Check whether your embedding model supports it before doing anything cleverer.
• Product quantisation — split the vector into sub-vectors and quantise each against a learned codebook. Used inside FAISS/IVF-PQ for large-scale search.
• Binary / scalar quantisation — float32 to int8 or 1 bit. Dramatic memory savings, small recall loss, often paired with an exact rescoring pass over the top candidates.

See ../09_rag_embeddings/04_vector_databases.md.

Interview angle

• “What does PCA do?” — finds orthogonal directions of maximum variance via eigendecomposition of the covariance matrix (equivalently SVD of centred data) and projects onto the top k. It’s linear and unsupervised.
• “Do you scale before PCA?” — yes, unless features already share units. PCA maximises variance, so unscaled features let unit choice decide which components dominate.
• “PCA vs t-SNE?” — PCA is linear, deterministic, invertible, and produces features you can use downstream. t-SNE is non-linear, stochastic, visualisation-only, and can’t embed new points. Different jobs entirely.
• “Can I use t-SNE output as model features?” — no. Non-deterministic, no transform for new data, and it distorts global distances. Use PCA for features.
• “How do you pick the number of components?” — cumulative explained variance (95% is a common target), the scree-plot elbow, or downstream validation performance if PCA is a preprocessing step.
• “Two clusters are far apart in my t-SNE plot. Meaningful?” — no. t-SNE preserves local neighbourhoods, not global distances; inter-cluster distances and cluster sizes in the plot don’t carry reliable meaning. Vary perplexity and check the structure persists.
• “How would you cut vector storage costs in a RAG system?” — Matryoshka truncation if the embedding model supports it, then quantisation (scalar or product) with an exact rescoring pass over top candidates to recover recall.