Mixture of Experts

MoE is the default for scaling in 2026. DeepSeek-V4, Llama 4, Kimi K2 and most large open releases are sparse — hundreds of billions to over a trillion total parameters with only a fraction active per token. If you describe frontier models as dense, that dates you.

The idea

Replace the MLP in each transformer block with N parallel expert MLPs plus a router that selects a few per token.

class MoELayer(nn.Module):
    def forward(self, x):
        logits = self.router(x)                      # (tokens, n_experts)
        weights, idx = logits.softmax(-1).topk(self.k, dim=-1)
        out = torch.zeros_like(x)
        for i in range(self.k):
            for e in range(self.n_experts):
                mask = idx[:, i] == e
                if mask.any():
                    out[mask] += weights[mask, i, None] * self.experts[e](x[mask])
        return out

Routing is per token, not per sequence. Different tokens in the same sentence go to different experts, and the routing changes at every layer.

Total vs active parameters

The distinction that matters, and the one interviewers check:

• Quality tracks total parameters, roughly — more experts means more stored knowledge.
• Compute cost tracks active parameters — you only run the selected experts.
• Memory tracks total parameters — every expert must be resident even if rarely used.

So MoE buys more capacity per FLOP, and pays for it in VRAM. That trade is the whole reason it’s dominant: inference cost is dominated by compute and bandwidth per token, while GPU memory can be scaled out across devices.

The catch nobody mentions first

You still have to hold all the weights. A 671B MoE with 37B active is cheap to run per token but needs enough aggregate GPU memory for 671B parameters. It’s not “a 37B model” — a common and revealing misstatement.

That’s why MoE suits datacentre serving with expert parallelism across many GPUs, and suits local/edge deployment badly.

Load balancing

Left alone, the router collapses: a few experts get most tokens, learn faster, get chosen more, and the rest are dead weight. Training adds an auxiliary load-balancing loss penalising uneven expert usage.

loss = task_loss + alpha * load_balance_loss

Newer approaches use auxiliary-loss-free balancing — adjusting per-expert routing biases directly — since the auxiliary loss slightly degrades quality by fighting the task objective.

Expert capacity caps how many tokens one expert accepts per batch. Overflow tokens get dropped (skipping the MLP, passing through the residual) or rerouted. Capacity that’s too tight loses tokens; too loose wastes memory.

Shared experts

A refinement in recent designs: one or two experts that every token uses, alongside the routed ones. The shared expert absorbs common patterns so routed experts can specialise, which reduces redundancy across experts. Used in the DeepSeek line.

Serving implications

Worth raising unprompted in a system-design conversation:

• Expert parallelism. Experts distribute across GPUs, so each token’s routing implies cross-device communication. Network becomes a bottleneck in a way dense models don’t have.
• Batch composition matters. In a batch, tokens scatter across many experts, so most experts activate anyway — MoE’s FLOP savings are strongest at small batch sizes and erode as batches grow.
• Load imbalance at inference causes stragglers: one overloaded expert stalls the step.
• Fine-tuning is harder. Routing can shift during fine-tuning, destabilising the balance learned in pretraining.

MoE vs dense

For most application engineers this is a model selection consideration rather than something you build. What you need is: know what the numbers mean, know the memory implication, and don’t call a sparse model by its active parameter count.

Interview angle

• “What is a Mixture of Experts model?” — the MLP in each block is replaced by many expert MLPs plus a router that picks a few per token. Total parameters grow while per-token compute stays low, so you get more capacity per FLOP.
• “A model has 671B total and 37B active. What does that mean for deployment?” — compute per token is like a 37B model, but you must hold all 671B parameters in memory. It’s a datacentre model with expert parallelism, not a 37B model you can run locally.
• “Why does MoE need a load-balancing loss?” — without it, routing collapses onto a few experts that improve fastest and get chosen more, leaving the rest untrained. The auxiliary loss penalises uneven usage; newer designs adjust routing biases directly to avoid the quality cost.
• “Is routing per sequence or per token?” — per token, and independently at each layer. Different tokens in one sentence take different expert paths.
• “When does MoE’s efficiency advantage shrink?” — at large batch sizes. Tokens scatter across experts, so most experts activate anyway and the FLOP saving erodes. The advantage is clearest at low batch sizes and for capacity per parameter served.
• “What’s a shared expert?” — an expert every token passes through, alongside the routed ones. It absorbs common patterns so the routed experts can specialise rather than each relearning the basics.