Polars — Lazy, Eager, and Expression API

7 interview angles 5 min read source

Polars — Lazy, Eager, and Expression API

The modern Pandas alternative. Rust core, multi-threaded, Arrow-native, lazy-optional. 5-30× faster than Pandas on typical workloads, with cleaner API for production pipelines.

Eager vs Lazy

Polars has two execution modes:

Eager — like Pandas

import polars as pl

df = pl.read_csv("orders.csv")
filtered = df.filter(pl.col("status") == "completed")
grouped = filtered.group_by("user_id").agg(pl.col("amount").sum())
sorted_df = grouped.sort("amount", descending=True)

Each operation runs immediately. Similar to Pandas semantics.

Lazy — query-planning

result = (
    pl.scan_csv("orders.csv")           # scan = lazy read
    .filter(pl.col("status") == "completed")
    .group_by("user_id")
    .agg(pl.col("amount").sum())
    .sort("amount", descending=True)
    .collect()                          # execute now
)

scan_* returns a LazyFrame. Operations build a query plan; .collect() triggers execution with optimization (predicate pushdown, projection pushdown, common subexpression elimination).

For files: lazy reads only the columns you actually use and only the rows that pass filters — huge wins.

Rule: use lazy for file-based and pipeline workloads. Use eager for quick interactive exploration.

Expression API

Polars’ killer feature. Operations are expressions; expressions compose; the engine optimizes them.

df.with_columns(
    pl.col("amount").sum().over("user_id").alias("user_total"),
    (pl.col("amount") / pl.col("amount").sum().over("user_id")).alias("share"),
    pl.when(pl.col("amount") > 100).then("high").otherwise("low").alias("tier"),
)

Vs Pandas’ patchwork of .groupby().transform(), .apply(), np.where() — Polars has one consistent expression language.

Common expression operators

pl.col("a")                                  # column reference
pl.col("a", "b")                             # multiple columns
pl.col(pl.Int64)                             # all columns of a type
pl.col("^prefix.*$")                         # regex column selector

pl.col("amount").sum()                       # aggregate
pl.col("amount").mean()
pl.col("amount").quantile(0.95)
pl.col("amount").rolling_mean(window_size=7) # window function
pl.col("amount").cum_sum()                    # cumulative

pl.col("name").str.contains("foo")           # string
pl.col("name").str.to_lowercase()
pl.col("ts").dt.year()                       # datetime
pl.col("ts").dt.weekday()

pl.when(...).then(...).otherwise(...)         # conditional
pl.struct(["a", "b"])                         # combine columns

Window functions

df.with_columns(
    pl.col("amount").sum().over("user_id").alias("user_total"),
    pl.col("amount").rank(method="dense").over("user_id").alias("rank_in_user"),
)

.over(...) is the Polars window function. Like SQL’s PARTITION BY. Same expression language; no separate API.

Reading data

# Eager
df = pl.read_parquet("file.parquet")
df = pl.read_csv("file.csv")
df = pl.read_json("file.json")
df = pl.read_database("SELECT * FROM users", connection_uri="postgres://...")

# Lazy (for files only)
ldf = pl.scan_parquet("file.parquet")
ldf = pl.scan_csv("file.csv")
ldf = pl.scan_ndjson("file.json")

Reading partitioned datasets:

ldf = pl.scan_parquet("s3://bucket/orders/*.parquet")     # glob
ldf = pl.scan_parquet("orders/", hive_partitioning=True)   # year=2026/month=5/...

hive_partitioning=True enables partition pruning — filter on year or month and only the matching partitions are read.

Joins

left.join(right, on="key", how="inner")        # default
left.join(right, on="key", how="left")
left.join(right, on="key", how="outer")
left.join(right, on="key", how="semi")          # rows in left that have a match in right
left.join(right, on="key", how="anti")          # rows in left with NO match in right
left.join(right, left_on="user_id", right_on="id", how="inner")

# Join by struct of columns
left.join(right, on=["a", "b"], how="inner")

semi and anti are common in Polars but verbose in Pandas (which requires isin or merge + filter).

Aggregations

df.group_by("user_id").agg(
    pl.col("amount").sum().alias("total"),
    pl.col("amount").mean().alias("avg"),
    pl.col("amount").count().alias("n"),
    pl.col("status").value_counts().alias("status_counts"),
    pl.col("ts").max().alias("last_seen"),
)

Multiple aggregations in one .agg() call. Each expression is independent — Polars runs them in parallel.

Streaming engine

For data larger than RAM:

result = (
    pl.scan_parquet("huge.parquet")
    .filter(pl.col("year") == 2026)
    .group_by("country")
    .agg(pl.col("revenue").sum())
    .collect(streaming=True)
)

Streams chunks through the query plan; aggregations spill to disk if needed. Not all operations support streaming (joins are the limitation); Polars warns when a step falls back to eager.

For 50 GB data on a 16 GB machine, this still works.

DataFrame vs Series

Polars distinguishes columns (Series) and tables (DataFrame) like Pandas, but with stricter semantics:

Indices don’t exist by default (Polars is index-free, unlike Pandas’ RangeIndex / MultiIndex).
Operations between Series respect order, not labels.
No SettingWithCopyWarning — Polars is functional; transformations return new frames.

Comparison to Pandas

# Pandas
df.groupby("user_id")["amount"].agg(["sum", "mean"])

# Polars
df.group_by("user_id").agg([pl.col("amount").sum(), pl.col("amount").mean()])

# Pandas conditional
import numpy as np
df["tier"] = np.where(df["amount"] > 100, "high", "low")

# Polars
df = df.with_columns(pl.when(pl.col("amount") > 100).then(pl.lit("high")).otherwise(pl.lit("low")).alias("tier"))

# Pandas rolling
df["ma"] = df["price"].rolling(7).mean()

# Polars (eager)
df = df.with_columns(pl.col("price").rolling_mean(window_size=7).alias("ma"))

Roughly Pandas-equivalent operations. Migration mostly mechanical, but the expression API rewards rethinking operations into expressions.

Lazy execution plan inspection

ldf = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("status") == "completed")
    .group_by("user_id")
    .agg(pl.col("amount").sum())
)

print(ldf.explain())
# OPTIMIZED PLAN: shows pushdown of filter into the scan, etc.

Like SQL’s EXPLAIN. Confirms predicate pushdown happened.

ldf.explain(streaming=True) shows the streaming plan.

Arrow integration

Polars uses Apache Arrow as its in-memory format. Zero-copy interop with:

PyArrow
DuckDB
numpy (for many types)
Spark (via Arrow exchange)
Many other Arrow-aware tools

import pyarrow as pa
arrow_table = pa.table({"a": [1, 2, 3]})
polars_df = pl.from_arrow(arrow_table)
arrow_back = polars_df.to_arrow()

For workflows that bridge Python ML tooling (pandas, scikit-learn) and big-data tools, Arrow as the lingua franca is huge.

Common gotchas

pl.col vs pl.lit: pl.col("x") references a column; pl.lit("x") is a literal value. In when().then(pl.lit("high")), pl.lit is needed because "high" isn’t a column.
with_columns returns a new frame. Polars is functional; reassign.
Schema strict. Polars infers types and enforces them. A column inferred as Int64 won’t accept strings later.
null vs NaN: Polars uses null (Arrow null). NaN is a float value; null is missing. Pandas conflates them (np.nan for everything missing); Polars distinguishes.
group_by (Polars 1.0+) — was groupby in earlier versions. Old code uses both spellings.
Streaming engine limitations — joins, some window operations don’t stream yet. Use eager for those.

When Polars beats Pandas

Any production pipeline: faster, lower memory, lazy optimization.
Reading large CSV / Parquet: predicate + projection pushdown.
Joining large DataFrames: multi-threaded.
Out-of-core data: streaming engine.
Memory-constrained environments: more efficient layout.

When Pandas still wins

Interactive notebooks where Pandas idioms are second nature.
Integration with the wider Python ML ecosystem (most libraries accept Pandas first, Polars second).
Plotting (matplotlib / seaborn integrate cleaner with Pandas).
Tiny datasets where speed doesn’t matter.
Code maintained by Pandas-fluent team.

Interview angle

“Polars vs Pandas — when each?” — Polars for production pipelines (faster, lazy optimization, lower memory, streaming for big data). Pandas for interactive analysis, integration with the wider ML ecosystem, tiny datasets. Migration cost is mechanical; the expression API is the bigger learning curve.
“Why is Polars faster?” — Rust core, multi-threaded by default, Arrow-native (efficient memory layout), lazy execution with query optimization (predicate pushdown, projection pushdown), no GIL contention.
“What’s a LazyFrame?” — a deferred computation. Operations on a LazyFrame build a query plan; .collect() executes with optimization. Use for files (scan_* returns LazyFrame). Materialize only at the end.
“How does Polars handle data larger than RAM?” — streaming engine. .collect(streaming=True) processes chunks through the query plan; aggregations spill to disk if needed. Not all operators stream (joins are the limitation); check explain(streaming=True).
“What’s the expression API?” — Polars’ composable column expression language. pl.col("a").sum().over("user_id") is a window-aggregated reference; you build complex transformations by composing expressions. Replaces Pandas’ patchwork of .groupby().transform(), .apply(), np.where().
“Migrating from Pandas — what stays the same vs changes?” — read/write API similar (pl.read_parquet vs pd.read_parquet). groupby → group_by + expression .agg(...). df["col"] = ... → df.with_columns(...). np.where → pl.when().then().otherwise(). Mostly mechanical.
“How does Polars integrate with Pandas / NumPy / DuckDB?” — via Arrow. pl.from_pandas() / df.to_pandas(); pl.from_arrow() / df.to_arrow(). Most modern Python data tools speak Arrow as the interchange format.