Production AI Systems·Lesson 64 of 67·expert
Structured logging, tracing and observability
A print statement cannot be queried. Emit JSON events with a request id, time every stage as a span, record the fields that explain an answer — and never log the key or the prompt by accident.
In this lesson · 20 min
- →Emit structured JSON logs instead of formatted strings
- →Configure the logging module with a custom formatter
- →Correlate every event in a request with one id
- →Time stages as spans and record their outcome
- →Redact secrets and personal data before they reach a log
When an answer is wrong, the answer tells you nothing. What tells you something is the record of what was retrieved, what was sent, what came back and how long each part took.
Strings you cannot query
The first line is readable and useless: “what is the p95 latency for questions over 2,000 input tokens?” cannot be answered without parsing prose. The second is one field access away.
A JSON formatter
extra= is the whole technique: structured fields travel with the message instead of being
formatted into it.
One id per request
Without a correlation id, concurrent requests interleave into an unreadable stream:
Every line carries request_id, so one grep reassembles a single request out of a day of traffic.
In an async service the id belongs in a contextvars.ContextVar, so it follows the work without
being threaded through every function signature.
Spans
The yield fields is what lets a stage record what it found as well as how long it took — the
candidate count is only known inside the block.
What to record for an LLM call
Redaction
Sampling
Logging every token of every prompt is expensive and rarely useful. Log the metadata always, the bodies rarely — and always when something went wrong:
Hashing the request id rather than drawing randomly means the decision is stable — the same request logs the same way if it is retried, and two services in one trace agree about it.
Try it yourself
Write scrub(value): recursively redact any dict key in SECRET_KEYS, and replace email addresses and sk- keys inside strings.
Write the span context manager: yield a dict the block can add fields to, and append one record with name, status, ms and those fields to RECORDS — even when the block raises.
Check yourself
Why log JSON events rather than formatted message strings?
Check yourself
Which field most often explains a wrong answer in a RAG system?
Recap
- Emit JSON events with
extra=, not formatted strings — fields are queryable, prose is not. - Give every request an id and put it on every event; in async code keep it in a
ContextVar. - Wrap each stage in a span that records duration, status and what the stage found.
- For an LLM call record the model, prompt hash, token counts, cache reads,
stop_reason, retrieval fields, citation check, outcome and cost. - Redact secrets and personal data at the point of emission; prompts contain whatever a user pasted.
- Always log metadata, sample bodies deterministically by request id, and always keep the body when something failed.