Property-Based Testing for Python IaC with Hypothesis

Example-based tests check the cases you thought of; property-based tests generate the ones you did not. This guide, part of testing Python IaC under Python IaC fundamentals and strategy, uses Hypothesis to test the naming, tagging, and validation logic inside your infrastructure helpers.

Context

Most IaC bugs are not in the cloud calls — they are in the Python that computes names, CIDRs, and tags before the provider ever runs. That pure logic is ideal for property-based testing: assert invariants (a generated name is always DNS-safe, a subnet always fits its VPC) across thousands of inputs, complementing the mock-based unit tests you already run.

The mechanical difference from an example test is that you stop supplying inputs and start describing them. A strategy describes the shape of legal input — text of a given length, integers in a range, a CIDR block — and Hypothesis draws from it, runs your assertion, and keeps drawing. What makes this practical rather than merely noisy is shrinking: when an assertion fails, Hypothesis does not report the 400-character Unicode string that happened to break it. It repeatedly simplifies the failing input while the failure persists, and reports the smallest one that still fails — typically something like Falsifying example: test_name_is_dns_safe(service='0', env='dev'). A minimal reproducer is the difference between a bug report and a debugging session.

The second thing that makes it stick is the example database. Hypothesis writes every failure it finds into a local .hypothesis/examples directory and replays those inputs first on subsequent runs, so a bug found once is checked forever afterwards without anyone writing a regression test by hand. That is also why a green run after a red one is only meaningful if the database survived — a point that matters a great deal in CI and is easy to get wrong.

This is not fuzzing, despite the surface similarity. A fuzzer looks for crashes with no notion of correctness; a property test asserts a specific claim you have written down. That claim is the real artefact. "Any service name and environment produce a valid DNS label" is a sentence you can put in a code review, and formulating it usually finds the bug before the test runs — most IaC helpers turn out to have an invariant nobody had ever stated, which is exactly why they break on the one input nobody tried.

Where does this pay off in infrastructure code specifically? Three places, in rough order of value: naming functions, because cloud resource names have strict and differing character rules that a helper must satisfy simultaneously; address maths, because subnet allocation is arithmetic with an off-by-one at every boundary; and configuration merging, because tag and default merging is where precedence bugs hide silently until an audit finds an untagged resource.

Prerequisites

• Python 3.9+ with pytest and hypothesis>=6.0 installed
• A pure function under test — name/tag/CIDR logic with no cloud calls, no clock reads, and no randomness of its own
• A clearly stated invariant the function must always satisfy, written as a sentence before it is written as an assertion
• A .gitignore entry for .hypothesis/ and a CI cache that preserves it between runs

# CLI: confirm the tools are available
pytest --version && python -c "import hypothesis; print(hypothesis.__version__)"

The "pure function" requirement is the one that decides whether this technique applies at all. A helper that reads os.environ or calls datetime.now() will produce different output for the same input, and Hypothesis will eventually catch it — reporting Flaky: Hypothesis test_name_is_dns_safe produces unreliable results: Falsified on the first call but did not on a subsequent one, which reads like a tool bug and is in fact a design one. Push impure inputs up into parameters before writing the property; the refactor is usually two lines and improves the code independently.

Implementation

1. State the invariant and generate against it

Suppose a helper builds a resource name from a service and environment. The invariant: the result is lowercase, <= 63 characters, and contains only DNS-safe characters, for any inputs.

# test_naming.py — property test for a resource-name builder
# CLI: pytest test_naming.py
import re
from hypothesis import given, strategies as st
from naming import resource_name   # the pure function under test

safe = re.compile(r"^[a-z0-9-]{1,63}$")

@given(service=st.text(min_size=1, max_size=40),
       env=st.sampled_from(["dev", "staging", "prod"]))
def test_name_is_dns_safe(service: str, env: str) -> None:
    name = resource_name(service, env)
    # Invariant: whatever the input, the output is a valid DNS label.
    assert safe.match(name), f"unsafe name: {name!r}"

If resource_name ever emits an uppercase letter or an over-long string, Hypothesis reports the exact minimal input that broke it.

The generator here is deliberately hostile within a realistic envelope. st.text() will produce empty-ish strings, whitespace, emoji, right-to-left marks and characters that uppercase into two code points — every one of which is a plausible service name typed by a human into a config file, and every one of which breaks a naive f"{service}-{env}".lower(). Constraining env with sampled_from rather than free text reflects reality: environments come from a fixed set, and generating env="🙂" would only produce failures nobody needs to fix.

One more line is worth adding to any property that has already caught a bug:

# test_naming.py (continued): pin the historical failure permanently
# CLI: pytest test_naming.py -q
from hypothesis import example, given, strategies as st


@example(service="İstanbul-api", env="prod")   # uppercase dotted I → two chars
@given(service=st.text(min_size=1, max_size=40),
       env=st.sampled_from(["dev", "staging", "prod"]))
def test_name_is_dns_safe_with_regression(service: str, env: str) -> None:
    assert safe.match(resource_name(service, env))

@example runs that exact input before any generated one, on every machine, regardless of whether the example database survived. Use it for failures that reached production; leave the rest to the database.

2. Compose strategies for structured inputs

Naming is the easy case because the input is flat. Address maths is where property testing earns its keep, because the invariants are relational — subnets must fit inside the VPC, not overlap each other, and cover the requested count — and no set of hand-picked examples covers the boundaries.

# test_subnets.py — properties of a subnet allocator
# CLI: pytest test_subnets.py -q --hypothesis-show-statistics
import ipaddress
from typing import Tuple
from hypothesis import given, strategies as st
from netplan import allocate_subnets   # pure function under test


@st.composite
def vpc_and_count(draw: st.DrawFn) -> Tuple[ipaddress.IPv4Network, int]:
    second = draw(st.integers(min_value=0, max_value=255))
    prefix = draw(st.integers(min_value=16, max_value=20))
    count = draw(st.integers(min_value=1, max_value=8))
    vpc = ipaddress.ip_network(f"10.{second}.0.0/{prefix}", strict=False)
    return vpc, count


@given(vpc_and_count())
def test_subnets_fit_and_do_not_overlap(
    case: Tuple[ipaddress.IPv4Network, int]
) -> None:
    vpc, count = case
    subnets = allocate_subnets(vpc, count)

    assert len(subnets) == count
    assert all(s.subnet_of(vpc) for s in subnets)
    for a, b in zip(subnets, subnets[1:]):
        assert not a.overlaps(b), f"{a} overlaps {b} inside {vpc}"

A composite strategy is the right tool whenever the inputs are related — here the prefix length constrains how many subnets can fit, so drawing them independently would produce impossible cases. Drawing them together inside @st.composite keeps every generated case legal, which matters because the alternative, assume(), discards illegal draws and can starve the generator. Filter too aggressively and Hypothesis stops the test with FailedHealthCheck: It looks like your strategy is filtering out a lot of data, which is the tool correctly telling you the generator, not the function, is wrong.

Notice that these three assertions are properties of the problem, not of the implementation. They would hold for any correct allocator, which means the test survives a rewrite of allocate_subnets — the opposite of a snapshot test, which pins the current output exactly. Both have a place; the property test is the one that keeps its value when the code changes.

3. Test algebraic properties, not just shape

Some of the most valuable invariants are relationships between calls rather than facts about one output. Tag merging is the canonical IaC example: merging is expected to be idempotent, overrides must beat defaults, and the result must obey the provider's limits.

# test_tags.py — algebraic properties of a tag merger
# CLI: pytest test_tags.py -q
from hypothesis import given, strategies as st
from tagging import merge_tags   # pure function under test

tag_key = st.text(min_size=1, max_size=128).filter(lambda s: not s.startswith("aws:"))
tag_val = st.text(max_size=256)
tag_map = st.dictionaries(tag_key, tag_val, max_size=25)


@given(defaults=tag_map, overrides=tag_map)
def test_merge_is_idempotent(defaults: dict[str, str], overrides: dict[str, str]) -> None:
    once = merge_tags(defaults, overrides)
    twice = merge_tags(once, overrides)
    # Provider note: AWS rejects the reserved `aws:` key prefix outright, so the
    # strategy excludes it rather than asserting on a case the API forbids.
    assert once == twice


@given(defaults=tag_map, overrides=tag_map)
def test_overrides_win_and_limits_hold(
    defaults: dict[str, str], overrides: dict[str, str]
) -> None:
    merged = merge_tags(defaults, overrides)
    for key, value in overrides.items():
        assert merged[key] == value
    assert len(merged) <= 50
    assert all(len(k) <= 128 and len(v) <= 256 for k, v in merged.items())

Idempotence is worth singling out because it is the property IaC code most often violates and least often tests. A merger that appends rather than replaces looks correct on the first apply and accumulates duplicate or stale tags on every subsequent one — a slow-motion failure that only surfaces when a resource hits the 50-tag ceiling months later.

Verification

Run the suite; a green run means the property held across every generated case. Record the failing seed when one fails so it is reproducible in CI.

# CLI: run and, on failure, re-run the printed seed deterministically
pytest test_naming.py -q

A passing property test is easy to over-trust, because "100 examples passed" says nothing about whether those examples were interesting. The statistics flag is the antidote — it reports how many examples were generated, how many were discarded by filters, and how much time went into generation versus execution:

# CLI: see what the generator actually produced
pytest test_subnets.py -q --hypothesis-show-statistics
# Look for: "of which N were invalid" — a high number means the strategy is
# fighting itself and real coverage is far lower than the example count.

Two numbers matter in that output. A large invalid count means your generator is producing cases the test rejects, so the effective sample is much smaller than it looks. And a "typical runtimes" figure in the hundreds of milliseconds means the function under test is doing more than pure computation — usually a sign that a cloud call or a file read has crept into what was supposed to be a pure helper.

For a deterministic gate, pin the behaviour explicitly rather than relying on defaults:

# conftest.py — environment-specific Hypothesis profiles
# CLI: HYPOTHESIS_PROFILE=ci pytest -q
import os
from hypothesis import HealthCheck, settings

settings.register_profile("dev", max_examples=50, deadline=200)
settings.register_profile(
    "ci",
    max_examples=1000,
    deadline=None,                      # CI runners are slow and jittery
    suppress_health_check=[HealthCheck.too_slow],
    derandomize=True,                   # same inputs for the same code
)
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))

derandomize=True is the setting that makes a property test usable as a merge gate: without it, a pull request can pass and the identical commit can fail on main because different inputs were drawn. With it, the same code always sees the same examples, so a failure is attributable to the change rather than to luck. Keep the randomised, higher-volume run for a nightly job where a new failure is information rather than an interruption.

Gotchas & Edge Cases

Flaky properties hide real bugs. If a test only sometimes fails, the invariant is too weak or the function is non-deterministic; pin randomness and tighten the property.

Over-broad generators. Generating arbitrary Unicode may surface inputs your system will never see; constrain strategies to realistic inputs so failures are actionable.

Slow suites. Property tests run many cases; cap max_examples for fast feedback locally and raise it in nightly CI.

The deadline fires on a function that is not slow. Hypothesis times each example and aborts with DeadlineExceeded: Test took 312.45ms, which exceeds the deadline of 200.00ms when one exceeds the limit. On a cold import or a loaded CI runner the first example is routinely the slow one, which makes this look intermittent. Set deadline=None in the CI profile rather than raising the number, and treat a consistently slow example as evidence the function is doing I/O.

Assertions that restate the implementation. A property computed the same way the function computes it always passes and tests nothing — assert name == f"{service}-{env}".lower() merely duplicates the code. A useful property is expressed in terms of the requirement (matches a DNS label regex, fits inside the VPC) so that a wrong implementation and a wrong test cannot agree.

Filters instead of generators. Reaching for .filter() or assume() to reject illegal draws works until the rejection rate climbs, at which point Hypothesis reports FailedHealthCheck: It looks like your strategy is filtering out a lot of data and stops. Construct legal values with @st.composite or st.builds instead of generating everything and throwing most of it away.

A CI runner with no example database. Every fresh container starts with an empty .hypothesis/, so the failures found yesterday are not replayed today; the suite is only as reproducible as the seeds it happens to draw. Either cache the directory between runs or promote important failures to @example decorators, which live in the source and cannot be lost.

Operational Notes

Property tests behave differently from example tests in a pipeline, and the differences are worth designing for rather than discovering.

The workable arrangement is three tiers. Locally, a small max_examples keeps the feedback loop under a second — nobody will run a suite that takes a minute on every save. On a pull request, a derandomised run with a few hundred examples gives a stable, attributable signal. Nightly, a large randomised run with the example database cached goes looking for new failures, and anything it finds becomes either a fix or an @example regression the next morning.

Treat the discovered failures as an asset rather than an annoyance. Each one is a concrete input that broke a helper every other test agreed was correct, which makes it the highest-value regression test you will get for free. Committing the minimal reproducer as an @example is a two-line change that makes the bug permanently unrepeatable, and it documents the edge case for the next reader far better than a comment would.

Be honest about the boundary of the technique. Property testing covers the deterministic Python between your configuration and the provider call: naming, address maths, tag merging, policy predicates, config parsing. It says nothing about whether the resource graph is wired correctly, which is what mock-based tests are for, and nothing about whether the synthesized output matches what shipped last week, which is what snapshot testing CDKTF stacks with pytest covers. A suite that has all three is well defended; a suite that has property tests alone is testing its helpers thoroughly and its infrastructure not at all.

One structural habit makes the whole approach cheaper: keep the pure logic in a module with no Pulumi or CDKTF import at all. A naming.py that imports nothing from an IaC library can be property-tested in milliseconds, reused by both a Pulumi program and a CDKTF one, and reasoned about by anyone. The moment resource construction leaks into that module, the tests need a runtime, the generation slows down, and the deadline warnings start. The same separation is what makes typed configuration objects worth building, as described in Python typing for cloud resource definitions.

FAQ

Does this replace mock-based tests?

No — it complements them. Use property tests for pure logic and Pulumi mocks for resource wiring.

Can I property-test synthesized output?

You can assert invariants over CDKTF-synthesized JSON, but that overlaps snapshot testing; property tests shine on pure helpers.

How many examples are enough?

The default (100) catches most issues; raise it for critical invariants and lower it where feedback speed matters.

How do I reproduce a failure that only happened in CI?

Copy the Falsifying example: line from the log and paste it into an @example decorator on the test — that reruns the exact input everywhere, with no seed or database required. Hypothesis also prints a @reproduce_failure decorator for the same purpose, but it is tied to the library version, so @example is the durable form.

Should .hypothesis/ be committed to version control?

No. It is a machine-local cache of failing inputs, it changes on every run, and it would create constant merge conflicts. Cache it in CI so failures replay between builds, and promote anything worth keeping to an @example in the test file.

What is a good first property to write?

Idempotence of whatever function your code calls repeatedly — a tag merger, a config normaliser, a name sanitiser. It needs no new generator beyond the input type, it takes three lines, and in IaC helpers it fails surprisingly often.

Related

• Unit Testing Pulumi Programs with Mocks — the mock-based counterpart for resource logic.
• Testing Python IaC — the parent topic on testing strategy.
• Python Typing for Cloud Resource Definitions — types and properties together catch a class of IaC bugs.