backend / python core / stdlib / 03_functools.md

functools — higher-order function utilities

3 min read source

functools — higher-order function utilities

The most-used items: lru_cache, cached_property, partial, wraps, reduce, singledispatch.

lru_cache — memoization decorator

Caches results keyed by arguments. Bounded by maxsize (LRU eviction) or unlimited.

from functools import lru_cache

@lru_cache(maxsize=128)
def fib(n):
    if n < 2: return n
    return fib(n-1) + fib(n-2)

fib(100)   # fast — without cache, this is exponential
fib.cache_info()    # CacheInfo(hits=98, misses=101, maxsize=128, currsize=101)
fib.cache_clear()

Constraints:

  • Arguments must be hashable. lru_cache won’t work on list/dict args.
  • Cache lives as long as the function does — usually for module lifetime. Memory leaks if maxsize=None and inputs are unbounded.
  • Methods bound to instances cache the instance in the args, preventing GC. Use @cached_property instead.

@cache (3.9+) is shorthand for @lru_cache(maxsize=None).

cached_property — compute once per instance

from functools import cached_property

class Article:
    def __init__(self, content: str):
        self.content = content

    @cached_property
    def word_count(self) -> int:
        print("computing")
        return len(self.content.split())

a = Article("hello world foo bar")
a.word_count    # prints "computing", returns 4
a.word_count    # 4 (cached, no print)
del a.word_count  # uncache

Stored in the instance’s __dict__, so the cache lifetime matches the instance. Doesn’t work with __slots__ (no __dict__).

partial — preset arguments

from functools import partial

def request(url, method, headers=None):
    ...

get = partial(request, method="GET")
get("/api/users")   # equivalent to request("/api/users", method="GET")

# Useful for callbacks:
button.on_click(partial(handle_click, button_id=42))

partial returns a callable. It’s typed properly with ParamSpec/Concatenate (see typing/06_paramspec_concatenate.md).

wraps — preserve metadata in decorators

from functools import wraps

def log(fn):
    @wraps(fn)
    def wrapper(*args, **kwargs):
        print(f"calling {fn.__name__}")
        return fn(*args, **kwargs)
    return wrapper

@log
def add(a, b):
    """Add two numbers."""
    return a + b

print(add.__name__)    # 'add'           ← without @wraps, would be 'wrapper'
print(add.__doc__)     # 'Add two numbers.'

Always use @wraps in decorators. It copies __name__, __doc__, __module__, __wrapped__ from the wrapped function.

reduce — fold a sequence

from functools import reduce
import operator

reduce(operator.add, [1, 2, 3, 4])           # 10  (((1+2)+3)+4)
reduce(operator.mul, [1, 2, 3, 4], 1)        # 24
reduce(lambda acc, x: acc + [x*2], [1,2,3], [])    # [2, 4, 6]

Often a list comprehension or sum/math.prod/max is clearer. Use reduce for accumulator patterns that don’t have a built-in.

singledispatch — type-based function dispatch

Single-method dispatch (one argument) without classes:

from functools import singledispatch

@singledispatch
def describe(x):
    return f"unknown: {x}"

@describe.register
def _(x: int):
    return f"int: {x}"

@describe.register
def _(x: list):
    return f"list of {len(x)} items"

@describe.register(str)        # alternate syntax
def _(x):
    return f"string: '{x}'"

describe(42)         # 'int: 42'
describe([1, 2, 3])  # 'list of 3 items'
describe("hi")       # "string: 'hi'"
describe(3.14)       # 'unknown: 3.14'

Useful for replacing if/elif type chains. For multi-arg dispatch, use multipledispatch (third-party) or class hierarchies.

total_ordering — fill in comparison methods

from functools import total_ordering

@total_ordering
class Version:
    def __init__(self, major, minor):
        self.major, self.minor = major, minor

    def __eq__(self, other):
        return (self.major, self.minor) == (other.major, other.minor)

    def __lt__(self, other):
        return (self.major, self.minor) < (other.major, other.minor)

v1 = Version(1, 2)
v2 = Version(2, 0)
v1 < v2     #
v1 <= v2    # — generated from __lt__ and __eq__
v1 >= v2    #
v1 > v2     #

Define __eq__ + one of <, <=, >, >=, get the rest free. Slight performance cost vs hand-written.

Interview angle

  • “How does lru_cache work?” (Hash args, cache results, evict LRU.) “What are its limitations?” (Hashable args only; memory; method-on-instance cache prevents GC.)
  • “How would you decorate a function so its __name__ is preserved?” (@functools.wraps(fn).)
  • “Implement memoization without lru_cache” (Dict + closure.)