Encoding categorical features

Models consume numbers. How you turn “country” into numbers changes both accuracy and whether your pipeline survives a new category appearing in production.

The options

One-hot

from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(
    handle_unknown="infrequent_if_exist",   # survives unseen categories
    min_frequency=10,                       # rare categories folded into one bucket
    sparse_output=True,
)

handle_unknown is the parameter that decides whether your service throws at 3am when a new country code appears. The default raises. Always set it explicitly.

The dummy variable trap: with N categories you get N columns that sum to 1, which is perfectly collinear with the intercept. It matters for plain linear regression (the design matrix becomes singular) and doesn’t matter for regularised models or trees. drop="first" handles it. Knowing when it matters is the better answer than reflexively dropping.

One-hot is a poor fit for trees at high cardinality: it creates many sparse binary features, each carrying little signal, and a tree has to make many splits to isolate one category.

Ordinal

from sklearn.preprocessing import OrdinalEncoder

enc = OrdinalEncoder(
    categories=[["small", "medium", "large"]],   # explicit order when one exists
    handle_unknown="use_encoded_value", unknown_value=-1,
)

Correct when the categories genuinely have an order. For unordered categories it’s fine for trees (which only compare thresholds and can carve out any subset given enough splits) and wrong for linear models, which will read “France=1, Germany=2, Spain=3” as Spain being three times France.

Target encoding

Replace the category with the mean target for that category. Powerful with high cardinality, and it leaks by construction unless you’re careful.

from sklearn.preprocessing import TargetEncoder     # sklearn >= 1.3

enc = TargetEncoder(smooth="auto", cv=5)            # internal cross-fitting
X_enc = enc.fit_transform(X, y)

Two things make it safe:

• Out-of-fold computation — each row’s encoding comes from folds that exclude it. sklearn’s TargetEncoder does this internally, which is why you should prefer it to a hand-rolled groupby().transform("mean").
• Smoothing toward the global mean — a category with three rows shouldn’t get its raw mean. Shrink by sample count.

encoded = (count * category_mean + smoothing * global_mean) / (count + smoothing)

If you’re using CatBoost, its ordered target statistics solve this natively and usually better. See ../02_classical_ml/05_gradient_boosting.md, and 04_data_leakage.md for what goes wrong without these precautions.

Hashing

Map the category through a hash into a fixed number of columns. Unbounded cardinality, constant memory, no fitted state — so no “unseen category” problem at all.

from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=2**18, input_type="string")

The trade-off is collisions (two categories share a column) and no way to map back to the original value. Standard in ad-tech and streaming systems where the category space is effectively infinite.

Embeddings

Learn a dense vector per category, trained jointly with the model. This is how neural networks handle high-cardinality categoricals, and it’s what makes them competitive on relational tabular data.

nn.Embedding(num_embeddings=n_users, embedding_dim=32)

Useful when categories have latent structure worth learning (users, products, locations) and you have enough data. Overkill for a five-value column.

High cardinality: the decision

The practical ladder for something like zip_code with 40,000 values:

1. Use a model with native support — LightGBM or CatBoost. Simplest and usually best.
2. Target encoding with out-of-fold and smoothing.
3. Group into meaningful buckets — region instead of zip. Domain knowledge beats encoding cleverness.
4. Frequency encoding if the count itself is informative.
5. Hashing if the space is unbounded or streaming.
6. Embeddings if you’re already using a neural network.

One-hot is not on that list, deliberately.

The production concerns

Unseen categories. Decide the behaviour explicitly: map to an “unknown” bucket, use the global mean, or reject. Silent failure here is common.

Category drift. New values appear and old ones disappear. Monitor cardinality and the unknown-bucket rate as a data-quality signal.

Train/serve consistency. The encoder is fitted state. It must be versioned and shipped alongside the model — a mismatch silently reorders your columns. This is a strong argument for saving the whole Pipeline rather than the bare estimator.

import joblib
joblib.dump(pipeline, "model.joblib")     # encoder + model together

Missing values are a category. Often an informative one. Don’t impute them away without checking — “the field was blank” may be your strongest signal.

Interview angle

• “How would you encode a 40,000-value zip code column?” — not one-hot. Prefer a model with native categorical support (LightGBM/CatBoost), or out-of-fold target encoding with smoothing, or roll up to a coarser geography. Mention hashing for unbounded spaces.
• “What’s wrong with label-encoding an unordered category for a linear model?” — it fabricates an ordering and a magnitude. The model reads category 3 as three times category 1. Fine for trees, wrong for linear models.
• “Why is target encoding risky?” — each row’s encoding includes its own label unless computed out-of-fold, which leaks and produces excellent offline metrics that don’t hold up. Smooth by category count too, so rare categories don’t get extreme values.
• “A new category appears in production. What happens?” — whatever handle_unknown says, so set it deliberately. Hashing avoids the issue entirely; fitted encoders need an explicit unknown bucket, and the unknown rate should be monitored.
• “Do you need to drop one column when one-hot encoding?” — only when perfect collinearity is a problem: plain linear regression with an intercept. Regularised models and trees are unaffected.
• “Where does the encoder live at serving time?” — inside the serialised pipeline, versioned with the model. Refitting it separately, or reconstructing column order by hand, is a classic source of training/serving skew.