Activation functions

Small topic, frequently asked, and the answers connect directly to why deep networks became trainable.

The family

Why sigmoid lost

Two failures, and the first is the important one.

Vanishing gradients. Sigmoid’s derivative is s(1-s), which peaks at 0.25 and approaches 0 at both tails. Backprop multiplies these across layers:

0.25^10 ≈ 0.000001

Ten layers in and the gradient reaching the early layers is effectively zero, so they never learn. This is the reason deep networks were untrainable before ReLU. See ../00_math_foundations/03_calculus_optimization.md.

Not zero-centred. Outputs are all positive, so all gradients for a given weight vector share a sign, forcing zig-zag optimisation paths. Tanh fixed this and still saturated.

ReLU

relu(x) = max(0, x)

Derivative is exactly 1 for x > 0 — no shrinkage, no saturation on the positive side. It’s also trivially cheap, and it produces sparse activations (roughly half the units output zero).

Dying ReLU is the failure mode: a unit whose input is negative for every example has zero gradient forever and can never recover. Usually caused by too high a learning rate pushing a bias strongly negative.

Diagnose by checking what fraction of activations are zero:

dead_fraction = (activations == 0).float().mean(dim=0)
# consistently 1.0 for a unit means it's dead

Fixes: lower the learning rate, use Leaky ReLU (max(0.01x, x)) or ELU/GELU, and check initialisation.

GELU and the smooth family

GELU(x) = x * Phi(x)     # Phi = standard normal CDF

Instead of a hard gate at zero, GELU weights the input by the probability that a standard normal is below it — a smooth, probabilistic gate. Small negative values pass through slightly attenuated rather than being zeroed.

Why transformers use it: the smooth derivative gives better-behaved gradients than ReLU’s discontinuity, there are no dead units, and empirically it trains better at scale. BERT, GPT and most transformer work adopted it.

SiLU / Swish (x * sigmoid(x)) is very similar and often interchangeable.

SwiGLU

The current standard in large language models. It’s a gated variant — it splits the input, passes one half through an activation, and multiplies:

def swiglu(x, W, V, W2):
    return (F.silu(x @ W) * (x @ V)) @ W2

The multiplicative gate lets the network learn what to let through per-dimension, which is more expressive than a fixed elementwise function. The cost is an extra weight matrix, so implementations reduce the hidden dimension (typically to 2/3) to keep the parameter count comparable.

Used in Llama, PaLM, Mistral and most recent architectures. Naming it — and explaining that “gated” means a learned multiplicative mask, not a different curve — is a good currency signal in an LLM interview.

Softmax

def softmax(z):
    z = z - z.max()          # subtract the max for numerical stability
    e = np.exp(z)
    return e / e.sum()

Turns logits into a probability distribution. The max-subtraction is mandatory in a hand-rolled implementation — exp(1000) overflows to inf and you get nan.

Two properties worth knowing: it’s shift-invariant (which is why subtracting the max is safe), and dividing the logits by a temperature before applying it rescales the entropy of the output. See ../00_math_foundations/04_information_theory.md.

Softmax is for mutually exclusive classes. Multi-label needs independent sigmoids.

Choosing

The short version:

• Hidden layers, CNN: ReLU. Leaky ReLU if you observe dead units.
• Hidden layers, transformer: GELU, or SwiGLU if you’re following a modern LLM recipe.
• Output, binary: sigmoid, fused via BCEWithLogitsLoss.
• Output, multi-class: softmax, fused via CrossEntropyLoss.
• Output, multi-label: sigmoid per label.
• Output, regression: none.

Activation choice is rarely where your accuracy is hiding. Data, architecture and learning rate matter far more — worth saying if asked to tune one.

Interview angle

• “Why did ReLU replace sigmoid in hidden layers?” — sigmoid’s derivative peaks at 0.25 and saturates, so gradients vanish exponentially with depth. ReLU has derivative exactly 1 on the positive side, so gradients pass through undiminished. It’s also cheaper and induces sparsity.
• “What is dying ReLU and how do you detect it?” — a unit whose pre-activation is negative for all inputs has zero gradient permanently. Detect it by measuring the fraction of examples for which each unit outputs zero; a unit at 100% is dead. Usually caused by too high a learning rate.
• “Why do transformers use GELU rather than ReLU?” — GELU is smooth, gating the input by the normal CDF rather than hard-thresholding. No dead units, better-behaved gradients, and empirically better at scale.
• “What is SwiGLU?” — a gated activation: split the projection, pass one branch through SiLU, multiply elementwise, project back. The learned multiplicative gate adds expressiveness; the hidden dimension is usually shrunk to offset the extra matrix. Standard in current LLMs.
• “Why subtract the max in softmax?” — numerical stability. exp of a large logit overflows to infinity, producing nan. Softmax is shift-invariant so subtracting the max changes nothing mathematically.
• “Softmax or sigmoid on the output?” — softmax when exactly one class applies; independent sigmoids when any subset can. Using softmax for multi-label suppresses everything but the top prediction.