Feature stores and point-in-time correctness

The infrastructure answer to two problems that keep recurring: training/serving skew, and computing features “as of” a moment in the past. Worth knowing even if you never run one, because the concepts are what interviewers are testing.

The problems it solves

Training/serving skew. Your training feature user_7d_order_count was computed with a pandas groupby in a notebook. The serving version is a Redis lookup written by another engineer three months later. They differ subtly — a timezone, an inclusive boundary, a filter on cancelled orders — and the model quietly underperforms in production with no error anywhere.

Point-in-time correctness. To build a training row for a prediction made on 15 March, every feature must have its 15-March value. Most operational tables store current state:

-- users table
user_id | status   | updated_at
42      | churned  | 2026-07-01

That table cannot tell you what user 42’s status was in March. Joining it into training data gives every historical row the present value — which encodes the outcome. That’s leakage, and it’s structural rather than a coding slip. See 04_data_leakage.md.

Reuse. Five teams each compute “customer lifetime value” slightly differently. A feature store makes it one definition with one owner.

The architecture

   event log / warehouse
            |
      feature definition (one place)
        /              \
   offline store     online store
   (historical,      (latest value,
    point-in-time)    low latency)
        |                  |
    training            serving

The essential property: both paths derive from the same definition. Offline produces historical values with correct timestamps; online serves the current value fast.

Point-in-time joins

The mechanic worth being able to describe. Given a set of (entity_id, event_timestamp) rows, you need each feature’s value as of that timestamp — the most recent value strictly before it.

SELECT
    e.user_id,
    e.event_ts,
    e.label,
    f.order_count_7d
FROM events e
LEFT JOIN LATERAL (
    SELECT order_count_7d
    FROM user_features f
    WHERE f.user_id = e.user_id
      AND f.computed_at <= e.event_ts        -- never look ahead
    ORDER BY f.computed_at DESC
    LIMIT 1
) f ON true

Two details that matter:

• Strictly before, not before-or-equal, when the feature is computed from data that includes the event itself.
• A staleness bound. If the most recent value is from six months ago, that’s probably not the value serving would have returned. Feature stores let you set a TTL so stale joins produce null rather than a lie.

This “as-of join” is the core capability. Doing it by hand for twenty features across millions of rows is where people either give up or introduce leakage.

Feast

The common open-source choice, and the one to name if asked.

from feast import Entity, FeatureView, Field, FileSource
from feast.types import Int64
from datetime import timedelta

user = Entity(name="user", join_keys=["user_id"])

user_stats = FeatureView(
    name="user_stats",
    entities=[user],
    ttl=timedelta(days=7),                 # staleness bound
    schema=[Field(name="order_count_7d", dtype=Int64)],
    source=FileSource(path="s3://.../user_stats.parquet",
                      timestamp_field="computed_at"),
)

# Training - point-in-time correct by construction
training_df = store.get_historical_features(
    entity_df=labels_df,                   # must carry event_timestamp
    features=["user_stats:order_count_7d"],
).to_df()

# Serving - same definition, millisecond lookup
features = store.get_online_features(
    features=["user_stats:order_count_7d"],
    entity_rows=[{"user_id": 42}],
).to_dict()

The symmetry between those two calls is the entire value proposition.

Alternatives: Tecton (commercial, Feast’s authors), Databricks Feature Store, SageMaker Feature Store, Vertex AI Feature Store, or a hand-rolled Redis-plus-warehouse setup, which is what most teams actually run.

Do you need one?

Usually not. The honest answer, and a good one to give:

A feature store is real operational weight — another datastore, another sync path, another failure mode. For a single batch-scored model, a well-tested shared Python module that both the training job and the scoring job import gives you most of the skew protection for none of the cost.

The principle matters more than the product. One definition, used by both paths, with timestamps. You can honour that with a shared library and a warehouse.

Streaming features

The hard case: features that must reflect events from seconds ago — “transactions in the last 5 minutes” for fraud.

That needs a streaming pipeline (Kafka plus Flink or Spark Structured Streaming) writing to the online store, and the offline equivalent computed the same way over historical events. Keeping those two implementations consistent is the standard source of skew, which is why some stacks compute both from a single declarative definition.

See ../../backend/10_message_queues/stream_processing/01_stream_processing_fundamentals.md.

Interview angle

• “What is a feature store and why would you use one?” — a system that defines features once and serves them to both training (historically, point-in-time correct) and inference (current value, low latency). It exists to kill training/serving skew and to make as-of joins tractable.
• “What is point-in-time correctness?” — every feature in a training row must hold the value it had at that row’s event timestamp, not its current value. Joining current-state tables into historical rows leaks the outcome backwards.
• “How do you compute a point-in-time join?” — for each (entity, event_time), take the most recent feature value strictly before event_time, with a staleness bound so ancient values become null rather than silently wrong.
• “What is training/serving skew?” — the same feature computed by two different code paths that disagree. Prevent it with one shared definition; detect the residue with shadow deployment comparing offline and online feature values on live traffic.
• “Would you introduce Feast for a single batch model?” — no. A shared feature-computation module imported by both the training and scoring jobs gives the same protection without another datastore to operate. Reach for a feature store when several models or teams share features, or when you need real-time serving of historical aggregates.
• “Your operational table only has current state. How do you build training data?” — you can’t, correctly. You need an event log or change-data-capture history with timestamps. This is the argument for CDC into the warehouse, and it’s a prerequisite for honest historical features.