Deep learning
Enough to reason about transformers, since that’s where it leads. Every structural trick here — residuals, normalisation, activations — reappears in an LLM.
| # | File | The question it answers |
|---|---|---|
| 01 | 01_neural_network_basics.md | why depth needs non-linearity; the training loop’s four silent bugs |
| 02 | 02_activation_functions.md | why ReLU beat sigmoid; GELU and SwiGLU in modern LLMs |
| 03 | 03_training_deep_networks.md | NaN loss debugging, warmup, fp16 vs bf16, fitting a model in memory |
| 04 | 04_cnn.md | inductive bias, residuals, CNN vs Vision Transformer |
| 05 | 05_rnn_lstm.md | why they lost to transformers — which is the interview value |
| 06 | 06_normalization_and_residuals.md | BatchNorm vs LayerNorm, RMSNorm, pre-norm vs post-norm |
| 07 | 07_transfer_learning.md | feature extraction, LoRA/PEFT, and fine-tune vs retrieve |
The three answers worth having ready
Why residual connections matter. They give the gradient an identity path, and reduce each block to learning a difference from identity. They fixed the degradation problem where deeper networks had higher training error — an optimisation failure, not overfitting.
Why LayerNorm in transformers. Variable sequence lengths, small per-device batches, and one-token-at-a-time generation all make batch statistics unusable. LayerNorm depends only on the sample, so it behaves identically at any batch size and in both training and inference.
Fine-tuning teaches form, not facts. Knowledge baked into weights can’t be updated or cited. If the requirement is “answer from our documents”, that’s retrieval. Try prompt, then few-shot, then RAG, then fine-tuning.