Production AI Systems·Lesson 61 of 67·expert
Testing non-deterministic systems with pytest
The model is not the unit under test. Test everything around it deterministically with fakes and fixtures, assert properties rather than exact strings, and keep the statistical question separate.
In this lesson · 22 min
- →Separate what to unit-test from what to evaluate statistically
- →Write pytest tests, fixtures and parametrised cases
- →Inject a fake client so the model never appears in a unit test
- →Assert properties of an output instead of its exact text
- →Mark the tests that cost money, and keep them out of the default run
pytest runs in this page, so the test suites below really execute. What cannot run here is the model — and that turns out to be the whole lesson.
Two different questions
The rule that follows: never put a live model call in a unit test. It is slow, costs money, and fails for reasons unrelated to your change.
pytest
Five tests, no model, no network, a few milliseconds. pytest.raises asserts that a failure
happens and that it is the right one — a test that merely expects “an exception” passes when the
code raises NameError from a typo.
Fixtures and parametrisation
A fixture builds what a test needs; parametrize runs one test over many cases:
Ten test cases from three functions. When one fails, pytest names the parameter that broke, so you learn which status is mishandled rather than that “the test failed”.
Make the model injectable
Code that constructs its own client cannot be tested. Pass it in:
Look at what those tests actually check: that the document reaches the user message, that a parameter reaches the system prompt, that empty input short-circuits, that a refusal is handled. Every one is a real bug you can ship, and none of them needs a model.
Assert properties, not strings
For the output that genuinely is non-deterministic, assert what must always hold:
“Cites only supplied chunks”, “is a citation or a refusal”, “is under sixty words” — those hold for every correct answer, so they are testable. “Equals this exact sentence” is not.
Floats and markers
Markers let one suite serve two purposes. In CI the default job runs
pytest -m "not live" on every pull request, and a nightly job runs the rest with a key.
Try it yourself
Write a FakeClient that replies from a script, records every request, and raises a clear error when the script runs out.
Write classify(client, ticket): take the client as a parameter, send the ticket as a user message, return the lower-cased stripped reply — and return 'unknown' for empty input or a refusal, without calling the model for the empty case.
Check yourself
Why should a unit test never make a real model call?
Check yourself
What kind of assertion works on genuinely non-deterministic model output?
Recap
- Unit tests answer “is the code correct?”; evaluation answers “is the system good?”. Keep them apart.
- Never call a live model in a unit test. Pass the client in as a parameter — that seam is what makes the code testable at all.
pytest.raises(Error, match=...)checks the failure is the right one;parametrizeturns one test into a table.- For non-deterministic output, assert invariants: only supplied citations, length bounds, schema validity.
pytest.approxfor floats, and markers plus-m "not live"to keep the expensive tests out of the default run.