Information theory for ML

Small topic, high interview leverage: it’s where cross-entropy loss, perplexity, KL divergence and temperature sampling all come from. If you can connect those four, you sound like someone who understands why language models are trained the way they are.

Entropy: how surprising is this distribution

H(p) = -sum( p(x) * log p(x) )

Measured in bits with log2, nats with ln. It is the average number of bits needed to encode a sample — equivalently, how uncertain the distribution is.

import numpy as np

def entropy(p: np.ndarray) -> float:
    p = p[p > 0]                      # 0 * log 0 == 0 by convention
    return float(-(p * np.log2(p)).sum())

entropy(np.array([0.5, 0.5]))            # 1.0 bit  - maximum for 2 outcomes
entropy(np.array([0.99, 0.01]))          # 0.08 bits - nearly certain
entropy(np.array([0.25]*4))              # 2.0 bits - maximum for 4 outcomes

Uniform is maximum entropy; a spike is minimum. A model that outputs near-uniform probabilities over the vocabulary has learned nothing.

Cross-entropy: your loss function

H(p, q) = -sum( p(x) * log q(x) )

p is truth, q is your prediction. It is the cost of encoding data from p using a code optimised for q — so it’s minimised exactly when q == p.

For classification, p is one-hot, so the sum collapses to a single term:

loss = -np.log(q[correct_class])

That’s the entire cross-entropy loss. Note the shape of it: predict 0.9 for the right class and the loss is 0.105; predict 0.1 and it’s 2.303; predict 0.001 and it’s 6.9. Confident wrong answers are punished enormously, which is exactly what you want and also why a single mislabelled example can dominate a batch.

# Never do this - log of a tiny float underflows
loss = -np.log(softmax(logits)[target])

# Do this - frameworks fuse softmax and log for numerical stability
loss = torch.nn.functional.cross_entropy(logits, target)   # pass LOGITS, not probs

Passing already-softmaxed probabilities to cross_entropy is a classic bug: it applies softmax again, flattening your distribution and producing a model that trains slowly and badly.

KL divergence: distance between distributions

KL(p || q) = sum( p(x) * log(p(x)/q(x)) ) = H(p, q) - H(p)

So cross-entropy = entropy of the truth + KL divergence. Since H(p) is fixed by the data, minimising cross-entropy is minimising KL divergence. That’s the one-line link between “the loss we compute” and “matching the true distribution”.

Properties that get asked about:

• KL >= 0, and KL == 0 only when the distributions are identical.
• It is not symmetric: KL(p||q) != KL(q||p). It’s not a distance metric. Use Jensen-Shannon divergence if you need symmetry.

Where you’ll meet it in practice:

• RLHF / DPO: a KL penalty against the reference model keeps the fine-tuned model from drifting into gibberish while chasing reward.
• Knowledge distillation: the student minimises KL against the teacher’s soft output distribution.
• VAEs: the KL term regularises the latent space toward a prior.
• Drift detection: KL or PSI between the training feature distribution and today’s production traffic. See ../15_mlops_llmops/.

Perplexity: cross-entropy in units people quote

perplexity = exp(cross_entropy)      # natural log

It’s the effective number of equally-likely choices the model is deciding between at each token. Perplexity 1 means perfect prediction; perplexity equal to vocabulary size means the model is guessing uniformly.

import math
math.exp(2.0)    # 7.39 - the model is as uncertain as a fair 7-sided die

Two caveats that separate a real answer from a memorised one:

• Perplexity is only comparable within the same tokenizer and the same evaluation data. Different tokenizers split text differently, so per-token numbers aren’t commensurable across model families.
• Low perplexity does not mean useful. It measures next-token prediction on held-out text, not helpfulness, factuality or instruction-following. That’s why LLM evaluation moved to task benchmarks and LLM-as-judge — see ../13_evaluation/.

Temperature: reshaping the distribution at sampling time

def softmax_with_temperature(logits: np.ndarray, T: float) -> np.ndarray:
    z = logits / T
    z = z - z.max()                  # stability: shift before exp
    e = np.exp(z)
    return e / e.sum()

Dividing logits by T before softmax is exactly a change of entropy: low T sharpens (low entropy), high T flattens (high entropy).

Related samplers worth naming: top-k (keep the k most likely tokens) and top-p / nucleus (keep the smallest set whose cumulative probability exceeds p). Nucleus adapts to how peaked the distribution is, which is why it’s the usual default.

For anything you’re going to parse — JSON extraction, function calls, classification — set temperature to 0. Non-determinism in a structured-output path is a bug, not creativity.

Mutual information

I(X; Y) = H(X) - H(X|Y)

How much knowing Y reduces uncertainty about X. It is zero exactly when they’re independent. Used for feature selection — unlike correlation, it catches non-linear dependence:

from sklearn.feature_selection import mutual_info_classif
scores = mutual_info_classif(X, y)

Warning for interviews: a feature with suspiciously high mutual information with the target is more often leakage than signal. See ../03_feature_engineering/04_data_leakage.md.

Interview angle

• “Why is cross-entropy the loss for classification rather than MSE?” — it comes from maximum likelihood for a categorical distribution, its gradient doesn’t vanish when predictions are confidently wrong (MSE’s does, through the sigmoid/softmax derivative), and it penalises confident errors heavily. Minimising it is equivalent to minimising KL to the true distribution.
• “Relationship between cross-entropy and KL divergence?” — H(p,q) = H(p) + KL(p||q). H(p) is a constant of the data, so minimising cross-entropy is exactly minimising KL.
• “What is perplexity, intuitively?” — exp(cross-entropy); the effective number of equally-likely options the model is choosing among per token. Only comparable across models sharing a tokenizer and eval set.
• “Is KL divergence a distance?” — no. It’s non-negative and zero only for identical distributions, but it’s asymmetric and violates the triangle inequality. Use Jensen-Shannon if you need a metric.
• “What does temperature do?” — divides logits before softmax, rescaling the entropy of the output distribution. Zero for anything you parse; higher for diversity. Pair with top-p rather than top-k as a default.
• “How would you detect that production data has drifted?” — compare the current feature distribution against the training reference using KL divergence, population stability index, or a Kolmogorov-Smirnov test per feature, and alert on the ones that move.