AWS Glue

Serverless, Spark-based ETL plus the Glue Data Catalog — the shared metastore that Athena, Redshift Spectrum, and EMR all read. For a backend role this is a lighter-touch topic; know the components and where Glue fits versus Athena.

The components

The mental model: Crawler discovers → Catalog stores the schema → Job transforms → Athena/Redshift query.

Glue ETL jobs

A Glue job is Apache Spark under the hood (or a lightweight “Python shell” job for small non-Spark work). You write PySpark (or Scala) with Glue’s extensions.

import sys
from awsglue.context import GlueContext
from pyspark.context import SparkContext

glueContext = GlueContext(SparkContext.getOrCreate())

# Read from the catalog
orders = glueContext.create_dynamic_frame.from_catalog(
    database="raw", table_name="orders_json"
)

# Transform, then write as partitioned Parquet
glueContext.write_dynamic_frame.from_options(
    frame=orders,
    connection_type="s3",
    connection_options={"path": "s3://lake/orders/", "partitionKeys": ["year", "month"]},
    format="parquet",
)

DynamicFrame vs Spark DataFrame

Glue introduces the DynamicFrame — like a Spark DataFrame but schema-flexible: it tolerates inconsistent/evolving schemas (a field that’s sometimes a string, sometimes a number) without failing, and has Glue-specific transforms (ResolveChoice, Relationalize, etc.). You can convert .toDF() to a regular Spark DataFrame when you want standard Spark operations, and .fromDF() back.

Use DynamicFrame for messy/evolving source data; convert to DataFrame for the heavy transformation logic.

Job bookmarks

A job bookmark lets a job track what it’s already processed so a re-run only picks up new data — incremental ETL instead of reprocessing everything. Essential for “run this hourly over the new S3 files” pipelines. Watch out: bookmarks have edge cases (they key on file paths/timestamps); a misconfigured bookmark either reprocesses or silently skips data.

Glue vs Athena — where each fits

They’re complementary: Glue crawlers + jobs build and transform the data lake; Athena queries it. For simple “convert CSV → partitioned Parquet” you can actually use Athena CTAS instead of a Glue job — Glue earns its keep on heavier Spark transformations and orchestrated pipelines. See ../03_Amazon_Athena/.

When you’d touch Glue as a backend engineer

Mostly: your application data lands in S3 (via Kinesis Firehose, S3 events, exports), a Glue crawler catalogs it, a scheduled Glue job transforms it into a query-optimized layer, and analytics/BI runs on Athena. You’re usually a producer of the data, not the person writing Spark jobs — but knowing the pipeline shape matters for system-design questions.

For a data-engineering role this would be a deep topic; for a Python backend role, the component model + the Glue-vs-Athena distinction is enough.

Common gotchas

• Crawler over-partitioning — pointing a crawler at deeply-nested S3 paths can create thousands of partitions or mis-infer the schema. Be deliberate about the path structure.
• Job bookmark misconfiguration — either reprocesses everything (bookmark not enabled) or skips data (bookmark state confused by changed paths).
• DynamicFrame vs DataFrame confusion — heavy transformation logic is cleaner on a converted Spark DataFrame; DynamicFrame is for ingesting messy schemas.
• Using a Glue job for simple format conversion — Athena CTAS is often simpler and cheaper for “CSV → Parquet.”
• Cold start / job startup time — Glue Spark jobs have minutes of startup overhead; not for low-latency work.

Common Interview Questions

The original question bank for this topic (kept for self-quizzing):

• Basic: what Glue is, Glue vs traditional ETL tools, main components, serverless processing model.
• Data Catalog: how the catalog works, crawler configuration, table schemas/metadata, catalog versioning.
• ETL jobs: creating/configuring jobs, job parameters/arguments, job bookmarks, error handling + retries.
• Transformation: writing transformation logic, DynamicFrame vs Spark DataFrame, transformation contexts, complex transformations.
• Performance: job performance tuning, monitoring/metrics, timeout/memory config, cost optimization.
• Integration: S3 data lakes, RDS/Redshift, streaming sources, connection types.
• Advanced: scheduling/triggers, development endpoints, cross-account processing, monitoring/alerting.
• Security: encryption, IAM roles/permissions, VPC/network security, audit logging.
• Troubleshooting: common job issues, debugging/logging, failure recovery, production best practices.

Interview angle

• “What are the main components of Glue?” — Data Catalog (shared schema store), Crawlers (discover schema + partitions), ETL Jobs (serverless Spark transformations), Triggers/Workflows (scheduling + pipelines).
• “Glue vs Athena?” — Glue transforms data (Spark ETL) and populates the Data Catalog; Athena queries data (Trino SQL) and reads the Data Catalog. Complementary — Glue builds the lake, Athena queries it.
• “DynamicFrame vs Spark DataFrame?” — DynamicFrame is Glue’s schema-flexible structure that tolerates inconsistent/evolving source schemas and has Glue-specific transforms; convert .toDF() to a regular DataFrame for standard Spark logic. DynamicFrame for messy ingestion, DataFrame for the transformation work.
• “What are job bookmarks?” — state that lets a Glue job process only new data on re-runs (incremental ETL) instead of reprocessing the whole source. Key for scheduled pipelines over a growing S3 dataset.
• “What populates the Glue Data Catalog and who reads it?” — Crawlers (and CREATE TABLE statements) populate it; Athena, Redshift Spectrum, EMR, and Glue jobs all read it. It’s the Hive-metastore-compatible shared schema layer of the AWS analytics stack.