Protocol Buffers (Protobuf)

Google’s binary serialization format. Schema-required, language-neutral, ~3–5× smaller than equivalent JSON, and ~10× faster to parse. Designed for forward + backward compatibility (you can add fields without breaking old clients).

For gRPC, Protobuf is the serialization layer. But Protobuf is useful on its own (storage formats, message queues, file formats).

Anatomy of a .proto file

syntax = "proto3";                // proto3 — current major version
package user.v1;                  // namespace (avoids name collisions)

option go_package = "github.com/myorg/proto/user/v1;userv1";
option java_package = "com.myorg.user.v1";

message User {
  string id = 1;                  // tag number 1 — identifies the field on the wire
  string name = 2;
  string email = 3;
  int32 age = 4;
  bool is_active = 5;
  repeated string tags = 6;        // list
  map<string, string> metadata = 7;
  UserRole role = 8;
  google.protobuf.Timestamp created_at = 9;
}

enum UserRole {
  USER_ROLE_UNSPECIFIED = 0;
  USER_ROLE_ADMIN = 1;
  USER_ROLE_EDITOR = 2;
  USER_ROLE_VIEWER = 3;
}

Scalar types

int32 and int64 are the workhorses. sint* is more efficient for numbers that are often negative (varint encoding pitfall). bytes for raw binary.

Tag numbers — the key to wire format

message User {
  string id = 1;
  string name = 2;
  string email = 3;
}

The numbers 1, 2, 3 are tags. On the wire, Protobuf encodes (tag, type, value) triplets. The field name is NOT in the wire format — only the tag.

Rules:

• Tag numbers are part of the schema contract. Renaming a field is safe (wire format unchanged); changing its tag number is breaking.
• Tags 1–15 use 1 byte; 16–2047 use 2 bytes. Reserve 1–15 for fields you access often.
• Don’t reuse tag numbers when you delete a field. Reserve them:

message User {
  reserved 4, 5;
  reserved "old_field_name";
  string id = 1;
  // ...
}

Reservations prevent accidental reuse, which would silently misinterpret old data.

proto3 defaults — no nulls

In proto3, fields have default values, not nullability. Missing fields decode to the type’s zero value:

This is a famous proto3 wart. “Did the client send is_active=false or omit it?” — same wire format, indistinguishable.

Workarounds:

• Use wrapper types: google.protobuf.BoolValue instead of bool — gives you nullable bool.
• Use proto3’s optional keyword (re-added later):

message User {
  optional bool is_active = 5;       // distinguishable from default
}

In Python, user.HasField("is_active") works only with optional or message-type fields.

repeated and map

message Post {
  string title = 1;
  repeated string tags = 2;                          // list of strings
  map<string, int32> view_counts_by_day = 3;          // dict
}

post = Post(title="Hello", tags=["python", "grpc"])
post.tags.append("backend")
post.view_counts_by_day["2024-01-15"] = 42

map is sugar over a repeated message of (key, value). Order isn’t preserved (it’s a hash map).

Enums

enum Status {
  STATUS_UNSPECIFIED = 0;    // proto3 requires 0 to be the zero value
  STATUS_ACTIVE = 1;
  STATUS_INACTIVE = 2;
  STATUS_DELETED = 3;
}

Convention: prefix values with the enum name (avoids namespace collisions in C/C++). First value MUST be 0.

Enum evolution: adding values is safe; old clients see unknown values as the integer value.

Nested messages

message Order {
  message LineItem {
    string product_id = 1;
    int32 quantity = 2;
  }

  string id = 1;
  repeated LineItem items = 2;
}

Nested types are scoped: Order.LineItem in code. Use sparingly — flat is usually clearer.

oneof — sum types

message Event {
  string id = 1;
  oneof payload {
    UserCreated user_created = 2;
    UserDeleted user_deleted = 3;
    PostPublished post_published = 4;
  }
}

Exactly one of the listed fields is set. Smaller wire size than a flat union; nicer code generation (Python: event.WhichOneof("payload") returns the field name).

Well-known types

Google provides standard message types:

import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
import "google/protobuf/empty.proto";
import "google/protobuf/any.proto";
import "google/protobuf/struct.proto";       // arbitrary JSON-like
import "google/protobuf/wrappers.proto";     // BoolValue, Int32Value, etc.

message Event {
  google.protobuf.Timestamp occurred_at = 1;
  google.protobuf.Duration retention = 2;
  google.protobuf.Struct metadata = 3;
}

Map to native types in most languages (Python datetime, JS Date). Avoids reinventing.

Schema evolution — what’s safe

Safe (wire format compatible):

• Adding a new field with a new tag number.
• Removing a field (use reserved so its tag can never be reused).
• Renaming a field (the field name isn’t in the wire format).
• Changing a field’s default value.
• Adding new enum values.

Breaking:

• Changing a field’s tag number.
• Changing a field’s type (mostly).
• Reusing a deleted field’s tag number for a different field.
• Removing a value from an enum (parsers may error).

Old code reading new data: unknown fields are ignored (or preserved if the codegen supports it). New code reading old data: missing fields decode as defaults.

This forward+backward compatibility is one of Protobuf’s biggest wins over JSON (where field additions can break naive parsers).

Encoding — varint

Most numeric fields use varint encoding: small numbers take 1 byte; large numbers more.

0    → 0x00
1    → 0x01
127  → 0x7F
128  → 0x80 0x01
16383 → 0xFF 0x7F

Each byte’s high bit is a “continuation” marker. Saves bytes for the common case of small numbers (IDs, counts, ages).

For numbers that are often negative, varint encoding of int32 is inefficient (negative numbers get padded to 10 bytes). Use sint32 (zig-zag encoding) instead.

Generating Python code

pip install grpcio-tools

python -m grpc_tools.protoc \
    -I=./proto \
    --python_out=./generated \
    --grpc_python_out=./generated \
    proto/user.proto

Produces user_pb2.py (messages) and user_pb2_grpc.py (gRPC service stubs).

The -I flag is the search path for imports between .proto files.

For pure Protobuf (no gRPC), use protoc directly:

protoc --python_out=./generated proto/user.proto

Using generated code in Python

from generated.user_pb2 import User, GetUserRequest

user = User(id="42", name="Alice", email="a@b.com", tags=["python"])
# Equivalent to filling fields one at a time:
user.id = "42"
user.name = "Alice"
user.tags.append("python")

# Serialize
data = user.SerializeToString()    # bytes
print(len(data))                    # surprisingly small

# Deserialize
user2 = User()
user2.ParseFromString(data)
print(user2.name)                   # Alice

# To dict (for logging/debugging)
from google.protobuf.json_format import MessageToDict, MessageToJson
print(MessageToJson(user, preserving_proto_field_name=True))

MessageToDict and MessageToJson are slow — use only for debugging / boundaries, not hot paths.

Protobuf vs JSON Schema

JSON wins on debuggability and ubiquity. Protobuf wins on size/speed and the schema being the contract.

For internal high-throughput APIs: Protobuf. For configs and external APIs: JSON.

Common pitfalls

• Reusing tag numbers after removing a field — decodes corrupted data without erroring.
• Forgetting optional on scalars when you need to distinguish unset from default — silent bug.
• Changing field types between versions — sometimes safe (int32 ↔ uint32), often not.
• Using MessageToJson in hot paths — orders of magnitude slower than Protobuf binary.
• Missing reserved declarations after field removal — future you (or a colleague) reuses the tag.
• Confusing field names with tag numbers — renaming is safe; renumbering is breaking.

Common interview confusions

• “Protobuf is just binary JSON.” — different schema model, different encoding philosophy (tags, not field names), different evolution rules.
• “proto3 has nullable types like Python.” — by default, NO — scalars don’t distinguish unset from default. optional keyword (re-added) brings nullability back.
• “Tag numbers are like Python field defaults.” — they’re wire identifiers, NOT defaults. The default of int32 is always 0.

Interview angle

• “What is Protocol Buffers?” — Google’s binary serialization format. Schema-required (.proto file), language-neutral, ~3–5× smaller than equivalent JSON, ~10× faster to parse. Used by gRPC for wire format.
• “What are tag numbers in Protobuf?” — wire identifiers for fields. The wire format encodes (tag, type, value); field NAMES are not on the wire. Tag numbers are part of the contract; renaming a field is safe, renumbering is breaking.
• “How does Protobuf handle schema evolution?” — adding fields (new tags) is safe — old code ignores unknown fields; new code sees defaults for missing fields. Removing requires marking tag reserved to prevent reuse. Field name changes are safe; tag changes are not.
• “What’s the proto3 ‘no nulls’ gotcha?” — scalar fields don’t distinguish “unset” from “default value.” is_active=false looks identical to “not sent” on the wire. Fix with optional keyword (re-added to proto3) or google.protobuf.BoolValue wrapper.
• “What’s oneof?” — a sum type: exactly one of the listed fields is set. Smaller wire format than a flat union; cleaner generated code.
• “What’s reserved in Protobuf?” — declares that a tag number (or field name) cannot be reused. Prevents accidentally repurposing a removed field’s tag, which would silently misinterpret old data.