Hypothesis and property-based testing
Example-based testing: “for input X, output should be Y.” Property-based testing: “for any input matching strategy S, property P should hold.” Hypothesis generates the inputs.
The shift in mindset
# Example-based — you pick the inputs
def test_reverse_twice():
assert reverse(reverse([1, 2, 3])) == [1, 2, 3]
assert reverse(reverse([])) == []
# ... did you cover the case with duplicates? unicode strings? floats?
# Property-based — hypothesis picks
from hypothesis import given, strategies as st
@given(st.lists(st.integers()))
def test_reverse_twice(xs):
assert reverse(reverse(xs)) == xsProperty tests catch bugs example tests miss because Hypothesis tries weird inputs you wouldn’t think to write: empty list, single element, huge lists, lists with negative ints, lists with None if your strategy allows.
Common strategies
from hypothesis import strategies as st
st.integers() # any int
st.integers(min_value=0, max_value=100)
st.floats(allow_nan=False)
st.text() # any unicode string
st.text(alphabet="abc", min_size=1, max_size=5)
st.lists(st.integers()) # list of ints
st.dictionaries(st.text(), st.integers())
st.tuples(st.integers(), st.text())
st.from_regex(r"\d{3}-\d{4}", fullmatch=True)
st.datetimes()
st.uuids()
st.one_of(st.integers(), st.none())
st.booleans()
st.sampled_from([Color.RED, Color.GREEN, Color.BLUE])composite — build complex inputs
from hypothesis import strategies as st
@st.composite
def user_profile(draw):
name = draw(st.text(min_size=1, max_size=20))
age = draw(st.integers(min_value=0, max_value=120))
email = draw(st.from_regex(r"[a-z]+@[a-z]+\.com", fullmatch=True))
return User(name=name, age=age, email=email)
@given(user_profile())
def test_user(u):
assert u.age >= 0draw is the bridge — call it inside @composite to consume from sub-strategies.
assume — reject invalid cases
from hypothesis import assume, given, strategies as st
@given(st.integers(), st.integers())
def test_division(a, b):
assume(b != 0) # skip cases where b is 0
assert a / b * b == pytest.approx(a)assume rejects the current case and tries another. Don’t overuse it — Hypothesis gives up after too many rejections. Prefer to encode the constraint in the strategy itself when you can:
@given(st.integers(), st.integers().filter(lambda b: b != 0))
def test_division(a, b): ...Shrinking — Hypothesis’s killer feature
When a property fails, Hypothesis automatically shrinks the input to the smallest failing case.
@given(st.lists(st.integers()))
def test_my_buggy_sort(xs):
assert my_sort(xs) == sorted(xs)If my_sort breaks on [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5], Hypothesis tries removing elements, halving values, etc. — typical output: “minimal failing example: [1, 0].”
Cause is usually obvious from a 2-element list. From 11 elements, much harder.
@example — keep specific cases
@given(st.lists(st.integers()))
@example([]) # always run with empty list
@example([1]) # always run with single element
def test_my_sort(xs):
assert my_sort(xs) == sorted(xs)Useful for boundary cases you know matter, in addition to the random sample.
Stateful testing — RuleBasedStateMachine
For testing systems with state (a queue, a cache, a state machine):
from hypothesis.stateful import RuleBasedStateMachine, rule, invariant
class CacheTest(RuleBasedStateMachine):
def __init__(self):
super().__init__()
self.cache = LRUCache(capacity=3)
self.model = {} # what the cache should contain
@rule(key=st.text(), value=st.integers())
def set(self, key, value):
self.cache.set(key, value)
self.model[key] = value
@rule(key=st.text())
def get(self, key):
if key in self.model:
assert self.cache.get(key) == self.model[key]
@invariant()
def size_within_capacity(self):
assert len(self.cache) <= 3
TestCache = CacheTest.TestCaseHypothesis generates random sequences of rules and checks invariants after each. Finds bugs in concurrent/state-dependent code that no example-based test would.
Settings and profiles
from hypothesis import settings, HealthCheck
@given(st.lists(st.integers()))
@settings(max_examples=500, deadline=None, suppress_health_check=[HealthCheck.too_slow])
def test_thing(xs): ...Useful settings:
max_examples=500— try more cases (default 100).deadline=None— disable per-example timeout (CI on slow runners).derandomize=True— same seed every run, reproducible.
Profile for CI:
# conftest.py
from hypothesis import settings, Verbosity
settings.register_profile("ci", max_examples=1000, deadline=None)
settings.register_profile("dev", max_examples=10)
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))HYPOTHESIS_PROFILE=ci pytestWhen property tests catch bugs example tests miss
- Off-by-one errors at empty / size-1 / size-N boundaries.
- Encoding issues with weird Unicode (combining chars, surrogate pairs, RTL text).
- Floating point edge cases —
nan,inf,-0.0, denormals. - Round-trip identities —
parse(serialize(x)) == x. - Algebraic laws — commutativity, associativity, idempotence.
- State machine invariants — “size never goes negative” “no item lost” “no duplicates.”
Common pitfalls
- Flaky tests due to timing — use
derandomize=Truefor reproducible runs, or setdeadline=None. - Strategy too narrow — generates only “easy” inputs, misses real bugs. Widen.
- Strategy too broad — too many
assumerejections, Hypothesis gives up. Narrow. - Database in property tests — each iteration costs an INSERT. Use in-memory, or move DB tests to integration suite.
- Reusing state across iterations — module-scoped fixture mutated by property tests across hundreds of cases. Make function-scoped.
Interview angle
- Q: “What’s property-based testing?” — invariants over generated inputs vs handpicked examples.
- Q: “What’s shrinking and why does it matter?” — minimal failing example; debugging is night-and-day easier.
- Follow-up: “Give an example property for a
sortfunction.” —sorted(sort(xs)) == sort(xs),len(sort(xs)) == len(xs), multiset equal. - Follow-up: “Where does property testing struggle?” — when the property is hard to state, side-effectful systems (need stateful testing or model-based).