Delta Lake — ACID Tables on Object Storage

Parquet alone is just files. Delta Lake (Databricks, 2019, Apache 2.0 since 2022) layers a transaction log over Parquet to give you ACID transactions, schema evolution, time travel, and MERGE/UPSERT on object storage.

The problem

You have a Parquet table on S3. Concurrent writers:

• Both compute new files in parallel.
• Both list the directory; both think their write should be visible.
• Reader scans the directory in the middle of a write; sees partial / inconsistent state.

Plain Parquet on S3 has no transactions. Delta Lake (and Iceberg, Hudi) solves this with a metadata layer.

How it works

my_table/
    _delta_log/
        00000000000000000000.json   # initial schema + first add
        00000000000000000001.json   # next commit
        00000000000000000002.json
        ...
    part-00000-xxx.parquet
    part-00001-xxx.parquet
    ...

The _delta_log/ directory contains an ordered series of JSON files. Each is an atomic commit describing a change (add file, remove file, schema change). Readers see the table by replaying the log:

1. List the log files.
2. Reconstruct the current set of “active” files (added minus removed).
3. Read those Parquet files.

Concurrent writers attempt to commit by creating the next log file (...0003.json). Object storage’s “create if not exists” semantics (or a metastore lock) serialize commits. Conflicts are detected; one writer succeeds, the other retries.

The transaction log is the table. The Parquet files are storage for the data; the log determines which files are part of the table at each version.

Operations

Read

df = spark.read.format("delta").load("s3://bucket/my_table")
# Or via Delta-rs (Python without Spark):
from deltalake import DeltaTable
dt = DeltaTable("s3://bucket/my_table")
df = dt.to_pandas()

Polars supports Delta reads natively:

import polars as pl
df = pl.read_delta("s3://bucket/my_table")

Write

df.write.format("delta").mode("append").save("s3://bucket/my_table")
df.write.format("delta").mode("overwrite").save("s3://bucket/my_table")

Atomic: either the commit succeeds (new file in _delta_log/) or it doesn’t. Readers see a consistent snapshot.

MERGE / UPSERT

The killer feature:

from delta.tables import DeltaTable

target = DeltaTable.forPath(spark, "s3://bucket/users")
target.alias("t").merge(
    source=updates.alias("s"),
    condition="t.user_id = s.user_id"
).whenMatchedUpdate(set={
    "email": "s.email",
    "updated_at": "s.updated_at"
}).whenNotMatchedInsert(values={
    "user_id": "s.user_id",
    "email": "s.email",
    "created_at": "s.updated_at",
    "updated_at": "s.updated_at"
}).execute()

SQL-like:

MERGE INTO users AS t
USING updates AS s
ON t.user_id = s.user_id
WHEN MATCHED THEN UPDATE SET email = s.email
WHEN NOT MATCHED THEN INSERT (user_id, email) VALUES (s.user_id, s.email);

Atomically upserts the changes. Without Delta, you’d be reading the table, computing the new state in memory, writing the whole thing back, hoping nothing else writes concurrently.

Update / Delete

target.update(
    condition="user_id = 42",
    set={"email": "'new@example.com'"}
)

target.delete("status = 'inactive'")

Row-level updates and deletes — not natively possible on plain Parquet. Delta achieves this by rewriting the affected files.

Time travel

# Read by version
spark.read.format("delta").option("versionAsOf", 10).load(path)

# Read by timestamp
spark.read.format("delta").option("timestampAsOf", "2026-05-01").load(path)

Since the log is append-only, any past version is reconstructible. Use cases:

• “What did the table look like before yesterday’s bad write?”
• “Reproduce a model training run with the data as it was.”
• “Restore a deleted row.”

Time travel is bounded by your retention policy (default 30 days for log files; old Parquet files removed by VACUUM).

OPTIMIZE and Z-Order

Over time, many small files accumulate (each write produces files). Reading takes longer with many small files.

OPTIMIZE my_table
ZORDER BY (user_id, event_date);

OPTIMIZE compacts small files into target-size ones (default 1 GB). ZORDER (Z-order curve) co-locates rows likely to be queried together, improving predicate pushdown for multi-column filters.

Run periodically (daily, weekly) on append-only / merge-heavy tables.

VACUUM

VACUUM my_table RETAIN 168 HOURS;

Deletes Parquet files that are no longer referenced by the active log version (i.e., old versions of MERGE / UPDATE / DELETE).

RETAIN 168 keeps 7 days of history. Lower retention = less storage; bounded time-travel window.

Gotcha: if a query is still running on an old version of the table, VACUUM that drops its referenced files breaks it mid-query. Long-running readers + aggressive VACUUM = errors.

Schema evolution

ALTER TABLE my_table ADD COLUMN new_field STRING;

Or implicit on write:

df.write.format("delta").option("mergeSchema", "true").mode("append").save(path)

New columns become null for older rows. Stricter schema enforcement at write time (Delta refuses writes with mismatched schemas without mergeSchema=true).

For breaking changes (type changes, drops), there’s overwriteSchema=true — usually requires a full rewrite.

Constraints

ALTER TABLE my_table ADD CONSTRAINT amount_positive CHECK (amount > 0);

Delta Lake supports CHECK constraints. Writes that violate them fail.

For uniqueness, you typically use MERGE patterns (upsert by key) instead.

Streaming with Delta

Delta is both a batch and a streaming source/sink:

# Stream read
df = spark.readStream.format("delta").load("s3://bucket/orders")

# Stream write
query = (
    incoming_df.writeStream
    .format("delta")
    .option("checkpointLocation", "s3://bucket/checkpoints/orders")
    .start("s3://bucket/orders_aggregated")
)

Exactly-once semantics via checkpoints; idempotent commits. The “lakehouse” pattern: streaming + batch unified.

Delta vs Iceberg vs Hudi

Three competing table formats; all do similar things (ACID Parquet via metadata layer):

Choosing:

• Delta — Databricks shop; tightly integrated with their ecosystem.
• Iceberg — multi-engine (Trino, Snowflake, BigQuery External, Athena, etc.); partition evolution and branching are differentiators. Increasingly the open-format default in 2025+.
• Hudi — Uber’s stronghold; record-level updates; streaming-first.

For new lakehouse projects in 2025: Iceberg is winning on open-engine support. Delta still dominates Databricks-centric shops.

Python without Spark

delta-rs is a Rust-based Delta Lake library with Python bindings — read/write Delta without needing Spark:

from deltalake import DeltaTable, write_deltalake
import pandas as pd

# Write
df = pd.DataFrame({"id": [1, 2, 3]})
write_deltalake("s3://bucket/my_table", df)

# Read
dt = DeltaTable("s3://bucket/my_table")
df = dt.to_pandas()

# Time travel
dt = DeltaTable("s3://bucket/my_table", version=5)

Lighter dependency than Spark; good for application code or smaller pipelines.

Common gotchas

• Small files explosion. Each write creates files. Streaming + many writers = millions of tiny files. Run OPTIMIZE regularly.
• Long-running readers + VACUUM. Reader’s files dropped mid-query. Use sufficient RETAIN window or hold readers against an explicit version.
• MERGE on huge data is slow. Each MERGE potentially rewrites large parts of the table. Z-order on the merge key helps; partitioning by a frequent filter key helps.
• Schema evolution edge cases. Type changes need full rewrite. Renames aren’t supported directly (use a view layer).
• Concurrent writers conflict. Two writers race for the next log file; one wins, the other gets ConcurrentAppendException. Application code must retry.
• Listing-based reads slow on huge tables. Some engines list the _delta_log/ directory on every read. Use a _last_checkpoint file (auto-created) to bound the listing.

When NOT to use a lakehouse format

• Append-only event logs, no updates. Raw Parquet is simpler.
• Small datasets. Overhead of the transaction log isn’t worth it.
• Operational DB workloads (high-concurrency OLTP). Use Postgres / MySQL; lakehouses are analytics-focused.

Interview angle

• “What problem does Delta Lake solve?” — ACID transactions, schema evolution, MERGE/UPSERT, and time travel on Parquet files. Plain Parquet on S3 is just files; concurrent writers corrupt readers. The Delta transaction log makes the table.
• “How does the Delta log work?” — _delta_log/ directory with append-only JSON files. Each file is an atomic commit (add file, remove file, schema change). Readers replay the log to determine which Parquet files form the current table state. Concurrent writers serialize via object-storage atomic creates or a metastore lock.
• “What’s MERGE for?” — atomic upsert based on a match condition. Without MERGE, you’d read + recompute + write the whole table, hoping no one else writes meanwhile. Delta MERGE is the SQL-standard pattern.
• “What’s OPTIMIZE / Z-ORDER?” — compact small files into target-size; Z-order rearranges rows so frequently-co-queried columns cluster together, improving predicate pushdown. Run periodically (daily/weekly) on append-or-merge-heavy tables.
• “Delta vs Iceberg?” — both ACID-on-Parquet formats. Delta: Databricks-centric, deeply integrated with their tooling. Iceberg: open-engine (Trino, Snowflake, BigQuery, etc.), partition evolution, branching — winning the multi-engine race in 2025+. For new lakehouses: Iceberg by default unless committed to Databricks.
• “Can you use Delta without Spark?” — yes, delta-rs (Rust + Python bindings). Read/write Delta tables from Python application code. Lighter dependency than Spark for moderate workloads.
• “What’s time travel for?” — querying the table as of a past version / timestamp. Use cases: investigate yesterday’s bad write, reproduce ML training data, restore deleted rows, audit. Bounded by VACUUM retention.