Pandas — Fundamentals and Indexing

The dominant tabular-data library in Python. Senior interviewers don’t ask “what is Pandas” — they ask about SettingWithCopyWarning, performance, and when Pandas breaks down.

Series vs DataFrame

import pandas as pd

s = pd.Series([1, 2, 3], index=["a", "b", "c"])
df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})

• Series = 1-D labeled array. One dtype.
• DataFrame = 2-D labeled table. Columns can have different dtypes.

A DataFrame is essentially a dict of Series (each column is a Series sharing the same index).

The index is a labeled axis (default: RangeIndex(0, n)). Operations align on the index, not position. This is the #1 thing to internalize — most Pandas surprises come from index alignment.

Indexing — the four accessors

df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]}, index=["x", "y", "z"])

df["a"]              # column "a" → Series
df[["a", "b"]]       # selected columns → DataFrame
df["x":"y"]          # ROW slice by label (inclusive of both ends!)

df.loc["x"]          # row "x" → Series (by label)
df.loc["x", "a"]     # cell (x, a) = 1
df.loc["x":"y", "a"] # rows x..y, column a (label slice, inclusive end)

df.iloc[0]           # row 0 → Series (by position)
df.iloc[0, 0]        # 1
df.iloc[0:2, 0]      # rows 0..1 (position slice, exclusive end)

df.at["x", "a"]      # scalar access, faster than .loc for single cell
df.iat[0, 0]         # scalar access by position, faster than .iloc

The label-vs-position distinction is the most common interview question. Get it wrong, get bizarre bugs.

The SettingWithCopyWarning

df[df["a"] > 1]["b"] = 0    # WARNING — chained indexing

Why: df[df["a"] > 1] may return a view or a copy; Pandas can’t reliably tell. Setting on it may not propagate to df.

Fix: single-step .loc:

df.loc[df["a"] > 1, "b"] = 0

One indexing operation, unambiguous.

Copy-on-Write (pandas 2.0+)

Pandas 2.0 introduced opt-in Copy-on-Write semantics; enabled by default in pandas 3.0+:

pd.options.mode.copy_on_write = True

Under CoW, every indexing operation returns a logical copy. No more SettingWithCopyWarning — and no more “did this propagate?” confusion. New code should run with CoW on.

Index alignment

a = pd.Series([1, 2, 3], index=["x", "y", "z"])
b = pd.Series([10, 20, 30], index=["y", "z", "w"])
a + b
# x     NaN
# y    12.0
# z    23.0
# w     NaN

Aligned by index, not position. Missing indices on either side → NaN. Same applies to DataFrame ops, merge, concat, arithmetic.

If you don’t want alignment:

a.values + b.values    # raw numpy, position-based

dtypes

df.dtypes
# a    int64
# b    object

Common dtypes:

• int64, int32 (and unsigned variants).
• float64.
• bool.
• object — usually means strings (boxed Python objects, slow).
• string — newer dedicated string dtype (faster).
• category — for low-cardinality strings/ints; memory-efficient.
• datetime64[ns] — timestamps.
• Int64 (capital I) — nullable integer (allows NaN without coercing to float).

Memory optimization

df.memory_usage(deep=True)         # bytes per column
df["status"] = df["status"].astype("category")
df["count"] = df["count"].astype("int32")

category is huge for strings with repetition — store the unique values once, store an int code per row. 10-100× memory reduction common.

object dtype on strings is the silent memory eater. A 1M-row string column can be 100+ MB.

NaN handling

df.isna().sum()                    # NaN count per column
df.dropna()                        # drop rows with any NaN
df.dropna(subset=["a"])            # drop rows with NaN in column a
df.fillna(0)                       # fill all NaN with 0
df.fillna(method="ffill")          # forward fill
df.fillna({"a": 0, "b": "unknown"}) # per-column fill

NaN is technically a float; mixing with int columns coerces to float. Use nullable dtypes (Int64, string, boolean) to keep NaN without coercion:

df["count"] = df["count"].astype("Int64")    # nullable int, NaN allowed

Reading and writing

pd.read_csv("file.csv", dtype={"id": "int32"}, parse_dates=["created_at"])
pd.read_parquet("file.parquet")
pd.read_sql("SELECT * FROM users", conn)
pd.read_json("file.json")

df.to_csv("out.csv", index=False)
df.to_parquet("out.parquet")
df.to_sql("users", conn, if_exists="append", index=False)

For large CSVs, chunk:

for chunk in pd.read_csv("big.csv", chunksize=100_000):
    process(chunk)

For analytics workloads, read Parquet, not CSV. Faster, smaller, preserves dtypes, supports column projection.

Iteration anti-patterns

# BAD — slow, O(n) Python interpretation
for index, row in df.iterrows():
    df.at[index, "x"] = row["a"] + row["b"]

iterrows allocates a Series per row. For 100k rows, takes seconds where vectorized takes ms.

# GOOD — vectorized, runs in C
df["x"] = df["a"] + df["b"]

Use iterrows only as a last resort (mixed types, complex per-row logic that can’t be vectorized).

If you must iterate, itertuples() is ~10× faster than iterrows (returns named tuples, not Series).

apply

df["doubled"] = df["a"].apply(lambda x: x * 2)

Convenient. Still slow — calls Python per row. Use only when:

1. Vectorized equivalent doesn’t exist.
2. The function does something genuinely per-row that NumPy can’t.

Series.apply with a numpy ufunc is no faster than .values operation. Prefer vectorized:

df["doubled"] = df["a"] * 2          # vectorized — milliseconds
df["doubled"] = df["a"].apply(lambda x: x * 2)   # apply — seconds for big df

Vectorization rule

Most “I need to do X per row” operations have a vectorized form. Check:

• Arithmetic: df["x"] + df["y"]
• Boolean: df["x"] > 5
• String: df["s"].str.upper(), df["s"].str.contains("foo")
• Datetime: df["ts"].dt.year, df["ts"].dt.hour
• Conditional: np.where(df["a"] > 0, df["b"], df["c"])

When in doubt, search for “Pandas vectorized X.” If you can’t find one, apply or iterrows — but profile.

Common gotchas

• SettingWithCopyWarning ignored — silent data integrity bug. Use .loc to set; enable CoW.
• Chained indexing — df[mask][col] = value doesn’t propagate. Use df.loc[mask, col] = value.
• NaN in integer column — coerces to float. Use Int64 dtype.
• groupby keys not in result — .groupby("col", as_index=False) keeps grouping cols as columns.
• merge duplicating rows — one-to-many or many-to-many joins; check key uniqueness.
• object dtype slow — string operations on object are slow; convert to string or category.
• df.to_csv doesn’t preserve dtypes — round-trip via CSV loses int/categorical info. Use Parquet.
• In-place ops — df.sort_values(by="a", inplace=True) is discouraged; Pandas 3.0 deprecates many inplace= args. Just reassign: df = df.sort_values(...).

When Pandas breaks down

• Data > RAM — Pandas loads everything. Use chunking, Dask, or Polars (streaming engine).
• High-throughput row updates — Pandas isn’t a database. Use real storage.
• Massive groupby on huge data — Polars or DuckDB beat Pandas significantly.
• Need lazy evaluation — Polars / Spark / Dask defer execution; Pandas doesn’t.
• Concurrent updates — no concurrency safety. Single-threaded library.

Rule of thumb: Pandas works well up to ~1 GB in RAM; struggles at 10+ GB; broken at 100+ GB. Beyond ~1 GB, evaluate Polars or DuckDB before scaling Pandas vertically.

Interview angle

• “.loc vs .iloc vs .at vs .iat?” — .loc: by label, slice inclusive. .iloc: by integer position, slice exclusive. .at / .iat: scalar versions, faster for single-cell access.
• “What’s SettingWithCopyWarning and how do you fix it?” — chained indexing (df[mask][col] = value) may return a view or copy ambiguously; the set may not propagate. Fix: single-step .loc[mask, col] = value. Or enable Copy-on-Write (pd.options.mode.copy_on_write = True), which eliminates the ambiguity.
• “Why is iterrows slow?” — allocates a Series per row; Python-level per-row iteration. Vectorized ops run in C. For per-row work, prefer vectorization; if you must iterate, itertuples() is ~10× faster than iterrows.
• “How do you optimize memory in a 2 GB DataFrame?” — categoricals for low-cardinality strings; smaller int types (int32 vs int64); nullable dtypes for ints with NaN; Parquet for storage. df.memory_usage(deep=True) to find culprits.
• “NaN in an int column — what happens?” — coerces to float64 (since NaN is a float). Use nullable Int64 to preserve int semantics with NaN support.
• “When does Pandas break down?” — around 1-10 GB it slows; beyond that, OOM or unworkable. Switch to Polars (single-machine, faster, supports streaming), DuckDB (SQL on big data), or Spark/Dask (distributed).
• “Why prefer Parquet over CSV?” — typed columns (preserves int/datetime/categorical), columnar (read subset of columns), compressed (~5-10× smaller), faster to read/write. CSV is text-only and slow.