Convolutional networks

Less central than they were — transformers took much of vision too — but the inductive-bias argument is a standard interview topic, and CNNs still win on small datasets and edge deployment.

The convolution

Slide a small learned kernel across the input, computing a dot product at each position.

nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)

Two properties define why this works for images:

• Local connectivity — a pixel’s meaning depends on its neighbours, not on distant pixels. So each output looks at a small window.
• Weight sharing — the same kernel applies everywhere, giving translation equivariance: a cat detector works wherever the cat is.

That’s the inductive bias. A fully-connected layer on a 224×224×3 image would need 150,528 weights per output unit and would have to learn translation invariance from scratch. A 3×3×3 kernel has 27 weights and gets it structurally.

Output size

out = floor((in + 2*padding - kernel) / stride) + 1

padding = kernel // 2 with stride=1 preserves spatial size (“same” padding) — the usual choice so you control downsampling explicitly rather than losing pixels at every layer.

Parameter count

params = kernel_h * kernel_w * in_channels * out_channels + out_channels

A Conv2d(64, 128, 3) has 3*3*64*128 + 128 = 73,856. Independent of input resolution — the same kernel handles any image size, which is why CNNs generalise across resolutions in a way MLPs can’t.

Receptive field

How much of the input one output unit sees. Stacking 3×3 convolutions grows it linearly; each layer adds 2.

Two 3×3 convolutions have the same 5×5 receptive field as one 5×5 convolution, but use 2*(9*C²) parameters instead of 25*C² — fewer parameters and an extra non-linearity. That’s the VGG insight and why 3×3 became standard.

Dilated (atrous) convolutions expand the receptive field exponentially without extra parameters, which is why they’re used in segmentation.

Pooling and downsampling

nn.MaxPool2d(2)          # halve spatial dims, keep the strongest activation
nn.Conv2d(..., stride=2) # learned downsampling - now more common

Pooling gives approximate translation invariance (small shifts don’t change the output) and reduces compute. Modern architectures often prefer strided convolutions, which learn how to downsample rather than fixing the rule.

Global average pooling replaced flatten-then-dense at the head: average each channel to one number, giving a fixed-size vector regardless of input resolution and far fewer parameters.

The typical architecture

nn.Sequential(
    nn.Conv2d(3, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
    nn.Conv2d(64, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
    nn.MaxPool2d(2),
    # ... repeat, doubling channels as spatial dims halve
    nn.AdaptiveAvgPool2d(1),
    nn.Flatten(),
    nn.Linear(512, n_classes),
)

The pattern: spatial dimensions shrink, channel count grows. Early layers learn edges and textures; deeper layers combine them into parts and objects.

Convolution → normalisation → activation is the standard ordering.

Residual connections

The change that made very deep networks trainable:

out = F.relu(self.bn2(self.conv2(F.relu(self.bn1(self.conv1(x))))) + x)

Adding the input back gives the gradient an identity path to flow through, so it reaches early layers undiminished. It also means a block only needs to learn a residual — if identity is optimal, driving the weights to zero achieves it, which is much easier than learning identity explicitly.

Before residuals, deeper networks performed worse on training data, not just validation — an optimisation failure, not overfitting. That framing is the good answer to “why do residual connections matter”. Transformers use the same trick, which is why the concept transfers.

Efficient variants

Depthwise separable convolution is the one worth being able to explain: instead of one kernel touching all channels and all spatial positions at once, do spatial filtering per channel then combine channels with a 1×1. Same expressive job, a fraction of the multiplies. It’s why on-device vision is feasible.

CNNs vs Vision Transformers

The trade-off is inductive bias against data. A CNN’s built-in assumptions are a gift when data is scarce and a ceiling when it’s abundant — ViTs can learn better representations because they’re not constrained to locality, provided you have the data to teach them.

In practice, fine-tuning a pretrained model is the default for both, and the choice is often decided by what checkpoint is available. Hybrid architectures (ConvNeXt, convolutional stems on transformers) blur the line further.

Interview angle

• “Why convolutions instead of fully-connected layers for images?” — local connectivity plus weight sharing. It matches how images work (nearby pixels are related, features can appear anywhere), massively reduces parameters, and provides translation equivariance structurally instead of having to learn it.
• “Why stack two 3×3 convolutions instead of one 5×5?” — same receptive field, fewer parameters (18C² vs 25C²), and an extra non-linearity between them. That’s the VGG argument.
• “What do residual connections do?” — give gradients an identity path so they reach early layers, and let a block learn a residual rather than a full transform. They solved the degradation problem where deeper networks had higher training error, which was optimisation failure rather than overfitting.
• “What is a depthwise separable convolution?” — factorise a standard convolution into per-channel spatial filtering followed by a 1×1 channel mix. Roughly 8-9x fewer parameters for similar accuracy, which is why mobile architectures use it.
• “CNN or Vision Transformer?” — CNN with limited data, because its inductive biases substitute for examples. ViT with large data or strong pretraining, because it isn’t constrained to locality. In practice, fine-tune whatever good pretrained checkpoint exists.
• “Why global average pooling instead of flatten plus dense?” — it makes the head independent of input resolution and removes a very large parameter block, which reduces overfitting.