Support vector machines

Less used in 2026 than in 2010 — gradient boosting took tabular data, deep learning took text and images — but the kernel trick is a classic interview topic and SVMs still win on small, high-dimensional datasets.

The idea: maximum margin

Among all hyperplanes separating two classes, pick the one with the largest distance to the nearest points. Those nearest points are the support vectors, and they’re the only ones that matter — move any other point and the boundary doesn’t change.

That gives good generalisation intuitively: a boundary sitting as far as possible from both classes is more robust to new data than one grazing the training points.

Soft margin and C

Real data isn’t separable, so the hinge loss allows violations at a cost:

minimize  (1/2)||w||^2 + C * sum(hinge_loss)

As with logistic regression, C is inverse regularisation strength. Small C = strong regularisation.

The kernel trick

The part that’s actually asked about.

The SVM optimisation depends on the data only through dot products x_i . x_j. So if you want to work in a higher-dimensional space where the data becomes separable, you don’t need to compute that mapping — you just need a function that returns the dot product in that space.

K(x_i, x_j) = phi(x_i) . phi(x_j)

You never compute phi. For the RBF kernel phi maps to an infinite-dimensional space, and you still only evaluate a cheap function of the original vectors. That’s the trick: implicit high-dimensional mapping at the cost of a dot product.

from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(
    StandardScaler(),              # MANDATORY for SVM
    SVC(kernel="rbf", C=1.0, gamma="scale", probability=False),
)

gamma, and how it interacts with C

gamma controls how far a single training point’s influence reaches.

C and gamma must be tuned together — they interact strongly. The standard approach is a log-scale grid:

from sklearn.model_selection import GridSearchCV
import numpy as np

grid = GridSearchCV(
    model,
    {"svc__C": np.logspace(-2, 4, 7), "svc__gamma": np.logspace(-4, 1, 6)},
    cv=5, n_jobs=-1,
)

gamma="scale" (the default, 1/(n_features * X.var())) is a sensible starting point and adapts to your data.

Scaling is mandatory

RBF depends on Euclidean distance. A feature in the thousands dominates one in the units, so the kernel effectively ignores everything but the large-scale feature. Not scaling an SVM is not a minor inefficiency — it silently breaks the model.

This is the practical difference from trees, which don’t care about scale at all.

The cost problem

That quadratic-to-cubic training cost is why SVMs fell out of favour: they’re impractical past roughly 10⁴-10⁵ samples. Gradient boosting scales far better.

For large datasets with a linear kernel, use the specialised solver:

from sklearn.svm import LinearSVC          # liblinear - scales to large n
from sklearn.linear_model import SGDClassifier   # hinge loss, out-of-core

LinearSVC is not just SVC(kernel="linear") with a faster solver — it optimises a slightly different objective (squared hinge by default, penalises the intercept). Usually irrelevant, occasionally surprising.

Probabilities

SVMs output a signed distance from the hyperplane, not a probability. probability=True fits Platt scaling via internal cross-validation — which makes training roughly 5x slower and can produce predictions inconsistent with decision_function.

If you need probabilities, logistic regression is usually the better tool from the start.

Where SVMs still win

• Small n, large d. Text classification with thousands of features and hundreds of documents. A linear SVM is often still the best classical choice there.
• Clear margin problems, where classes genuinely separate.
• One-class SVM for novelty detection when you only have “normal” examples.

Where they lose: large datasets, mixed feature types, categorical features, needing calibrated probabilities, and needing interpretability (an RBF SVM is a black box).

Interview angle

• “What is the kernel trick?” — the optimisation depends on the data only through dot products, so replacing the dot product with a kernel function computes similarity in a higher-dimensional space without ever constructing that space. RBF corresponds to an infinite-dimensional feature space at the cost of one exponential.
• “What are support vectors?” — the training points on or inside the margin. They alone determine the boundary; removing any other point changes nothing. That’s also why inference cost scales with their number.
• “What do C and gamma do?” — C trades margin width against violations (inverse regularisation); gamma sets how far one point’s influence extends in RBF. High gamma plus high C overfits dramatically. Tune them jointly on a log grid.
• “Do SVMs need feature scaling?” — yes, unavoidably. RBF is distance-based, so unscaled features let the largest-magnitude one dominate the kernel. Unlike trees, this isn’t an optimisation detail — it breaks the model.
• “Why aren’t SVMs used much any more?” — training is quadratic-to-cubic in samples, so they don’t scale; they need scaling and careful tuning; they don’t produce calibrated probabilities natively; and gradient boosting beats them on tabular data while deep learning beats them on text and images.
• “When would you still pick one?” — small, high-dimensional data with a clear margin, especially linear SVM for text, or one-class SVM for novelty detection.