Spark — Architecture, DAGs, and Execution

The conceptual model behind PySpark. Senior interviews probe the mental model (driver, executors, shuffle) more than the API.

The components

[Driver]      ←→  [Cluster Manager (YARN / Kubernetes / standalone / Databricks)]
   │
   ├─→ [Executor 1] (JVM): tasks + cached data
   ├─→ [Executor 2] (JVM)
   ├─→ [Executor 3] (JVM)
   └─→ [Executor N] (JVM)

• Driver — the JVM running your main function / notebook cell. Holds the SparkContext / SparkSession. Builds the DAG of operations; orchestrates execution.
• Executors — worker JVMs on cluster nodes. Run tasks; hold data partitions in memory; cache RDDs/DataFrames between stages.
• Cluster manager — schedules executors onto machines. YARN (Hadoop), Kubernetes (modern), standalone, or Databricks’ managed equivalent.

PySpark adds a Python process per executor that runs your Python UDFs. Communication between the JVM executor and Python processes is via pipes — slow for fine-grained per-row UDFs.

RDD vs DataFrame vs Dataset

For Python, you use DataFrames in 99% of cases. RDDs are the lower-level escape hatch for operations the DataFrame API can’t express.

# DataFrame — the production default
df = spark.read.parquet("s3://.../orders/")
result = (
    df.filter(df.status == "completed")
      .groupBy("user_id")
      .agg(F.sum("amount").alias("total"))
      .orderBy(F.col("total").desc())
)

# RDD — only when you need it (custom partitioning, complex stateful ops)
rdd = df.rdd

Lazy evaluation

Spark operations are either:

• Transformations — lazy. Build the execution plan but don’t run. filter, select, groupBy, join, withColumn, …
• Actions — eager. Trigger execution. count(), show(), collect(), write.*, toPandas().

df = spark.read.parquet("orders.parquet")       # no I/O yet (lazy)
df_filtered = df.filter(df.status == "completed")   # still no I/O
df_grouped = df_filtered.groupBy("user_id").count()  # still no I/O

df_grouped.show()    # NOW Spark actually executes — reads, filters, groups

This is why a single typo in a long chain doesn’t surface until the action runs. The error message lives at the action, not the transformation that’s wrong.

The DAG

When you call an action, Spark:

1. Compiles your transformations into a logical plan.
2. Catalyst optimizer rewrites the plan (predicate pushdown, projection pushdown, join reordering).
3. Optimized plan becomes a physical plan.
4. Physical plan is divided into stages, separated by shuffles.
5. Each stage = multiple tasks (one per partition).
6. Tasks distributed across executors.

df.filter(...).select(...).groupBy("x").agg(...).join(other, "key").write(...)

Stages:
  Stage 1: read + filter + select (narrow ops, no shuffle)
  Stage 2: groupBy + agg (requires shuffle)
  Stage 3: join (requires shuffle)
  Stage 4: write

explain(True) shows the plan:

df.explain(True)
# == Parsed Logical Plan ==
# == Analyzed Logical Plan ==
# == Optimized Logical Plan ==
# == Physical Plan ==

Senior debugging: read the physical plan to spot accidental cartesian joins, missed predicate pushdown, broadcast-vs-sort-merge decisions.

Shuffles — the cost center

A shuffle is when Spark redistributes data across executors. Operations that shuffle:

• groupBy / agg (must collect rows with the same key on one executor).
• join (must collocate matching keys).
• repartition / coalesce (explicit).
• sort / orderBy (global ordering).
• distinct.

How a shuffle works:

1. Each executor partitions its data by the shuffle key.
2. Writes intermediate files to local disk.
3. Other executors fetch the files relevant to their partition.

This involves: disk I/O + network I/O + serialization + sort. Shuffles are 10-100× more expensive than narrow ops on the same data volume.

Spark performance tuning is largely “reduce or speed up shuffles.”

Narrow vs wide transformations

Narrow transformations are free (relative to the cost of reading the data). Wide transformations cost shuffle.

Partitioning

The unit of parallelism. A 100 GB dataset split into 1000 partitions = 1000 tasks executable in parallel (subject to executor cores).

Default: ~128 MB per partition for HDFS reads. Configurable via spark.sql.files.maxPartitionBytes.

Too few partitions: parallelism limited; executors idle. Too many partitions: scheduling overhead, small-file shuffle pain.

Rule: target 100-200 MB per partition for most workloads.

df.rdd.getNumPartitions()    # check partition count
df.repartition(200)          # full shuffle to N partitions
df.coalesce(50)              # reduce partition count without shuffle (only down)

coalesce is cheap; repartition is a shuffle.

Broadcast joins

When joining a large table with a small one, broadcast the small one to all executors — avoids shuffling either side.

from pyspark.sql.functions import broadcast

big_df.join(broadcast(small_df), "key")

Threshold for auto-broadcast: spark.sql.autoBroadcastJoinThreshold (default 10 MB). Below that, Spark auto-broadcasts. Above, it sort-merge-joins (shuffles both).

If the “small” side is < a few hundred MB, force broadcast via broadcast(...) — much faster than a full shuffle.

Sort-merge joins (default for large-large)

Both sides shuffled by join key, then sort-merged. Standard for joining two big tables. Expensive but correct.

Optimization: pre-partition both inputs by the join key so the shuffle is avoided:

df1 = df1.repartition(N, "key")    # bucket by key
df2 = df2.repartition(N, "key")
df1.join(df2, "key")               # uses existing partitioning, may avoid shuffle

Or use Spark’s bucketing feature at write time (df.write.bucketBy(...)) to pre-partition on disk.

Data skew

When a few keys dominate, the partitions holding them are huge. Symptom: one task takes 10× longer than others.

Diagnosis: look at the Spark UI’s “Tasks” tab. Sort by duration; if one task is way out, it’s skew.

Mitigations:

• Salt the key — append a random suffix to the heavy key, distributing across N partitions; join then aggregates back.
• Skew join hint — spark.sql.adaptive.skewJoin.enabled = true (AQE detects skew at runtime and re-splits).
• Two-stage aggregation — partial aggregate before final aggregate.

Adaptive Query Execution (AQE, Spark 3.0+) handles many skew cases automatically; enable it.

Catalyst optimizer

Spark’s SQL optimizer. Rewrites the logical plan:

• Predicate pushdown — push filters to the data source so less data is read.
• Projection pushdown — read only required columns.
• Constant folding — evaluate compile-time expressions once.
• Join reordering — try cheaper joins first.
• CBO (cost-based optimizer) — when statistics are available, pick the cheapest plan.

explain("formatted") shows the optimized plan vs your input.

df.write.mode("overwrite").parquet("...")    # collect stats during write
spark.sql("ANALYZE TABLE my_table COMPUTE STATISTICS FOR ALL COLUMNS")

Without statistics, Catalyst picks defaults — sometimes wrong (especially for joins).

PySpark UDFs — performance gotcha

from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

@udf(returnType=IntegerType())
def double_it(x):
    return x * 2

df.withColumn("doubled", double_it("amount"))

A regular UDF serializes each row from JVM → Python → JVM. Per-row overhead is huge. Avoid in hot paths.

Better: pandas UDFs (vectorized):

import pandas as pd
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import IntegerType

@pandas_udf(IntegerType())
def double_it(s: pd.Series) -> pd.Series:
    return s * 2

Batches rows into pandas Series; Arrow-based JVM↔Python transfer. 10-100× faster than regular UDFs.

Best: built-in Spark functions (no UDF at all):

df.withColumn("doubled", df["amount"] * 2)

Native Spark expressions are optimized by Catalyst; no JVM↔Python crossing.

Rule of thumb: built-ins > pandas UDF > regular UDF. Reach for UDFs only when no built-in fits.

Adaptive Query Execution (AQE)

Spark 3.0+ optional, default-on in newer versions. AQE re-plans at runtime based on actual statistics:

• Auto-adjust partition counts after shuffles based on actual data size.
• Convert sort-merge to broadcast when one side turns out small.
• Skew join handling — detect and split skewed partitions.

spark.conf.set("spark.sql.adaptive.enabled", "true")

Big win on real-world workloads where static optimization can’t predict data shape.

When NOT to use Spark

• < 10 GB data — Polars / DuckDB on one machine win. Spark overhead dominates.
• Interactive analytics on small data — DuckDB SQL is way faster.
• Latency-sensitive serving — Spark is batch / micro-batch. Use a real database.
• Operational simplicity matters — Spark cluster ops are non-trivial. Databricks / EMR / Glue manage some of this.

Spark wins at 100 GB+ where distributed compute is necessary.

Interview angle

• “What’s the driver vs executor split?” — Driver runs your code, builds the DAG. Executors are workers that run tasks. Driver doesn’t process data (other than collect() results); executors do.
• “Lazy evaluation — why?” — lets Catalyst see the whole plan and optimize (push filters down, prune columns, reorder joins). Eager evaluation would force premature decisions. Trade-off: errors surface only at actions, not at the transformation that’s wrong.
• “What’s a shuffle?” — data redistribution across executors, triggered by wide transformations (groupBy, join, sort, repartition). Each side writes partitioned files to disk; counterparts fetch. Network + disk + serialization — expensive. Most performance work is reducing or speeding up shuffles.
• “Broadcast vs sort-merge join?” — broadcast: send small side to every executor, join in memory. Sort-merge: shuffle both sides, sort, merge. Broadcast much faster when one side is small (~10 MB by default; force with broadcast() for slightly bigger).
• “How do you debug data skew?” — Spark UI Tasks tab; sort by duration; one task taking 10× others = skew. Mitigations: salt the heavy key, two-stage aggregation, AQE’s skew join handling.
• “PySpark UDF — why is it slow?” — regular UDFs serialize each row JVM → Python → JVM. Per-row Python overhead is huge. Use pandas UDFs (vectorized, Arrow transfer) when you need Python; built-in Spark functions (no UDF) when possible.
• “When is Spark overkill?” — under ~10 GB, Polars / DuckDB on one machine beat Spark. Spark startup overhead, cluster ops, and per-stage shuffle cost only pay off at large scale.