Production AI Systems·Lesson 65 of 67·expert
Serving models with FastAPI
Typed request and response models, async endpoints, streaming, dependency injection for the client, and error handling that turns an upstream 429 into the right status code.
In this lesson · 22 min
- →Define an endpoint with typed request and response models
- →Validate input at the boundary and return a useful 422
- →Inject the model client as a dependency so the app is testable
- →Map upstream failures onto correct HTTP status codes
- →Stream a response, and expose health and readiness checks
The smallest useful service
Everything typed in AskRequest is enforced before your function runs. Everything in
AskResponse is enforced on the way out, so a handler cannot quietly return a shape the client
does not expect.
Validation is free
Five cases, one model definition, no validation code. The 422 body names the field and the problem, which is what a client integrating against you needs.
Inject the client
The same seam as the testing lesson, expressed as a FastAPI dependency:
dependency_overrides is why this matters: the test swaps the client without touching the
handler, and production wiring stays in one function.
Error handling
An upstream failure must become the right status code, not a 500:
Streaming
For a chat endpoint, streaming is the difference between a spinner and visible progress:
import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
app = FastAPI()
@app.post("/ask/stream")
async def ask_stream(body: AskRequest):
async def events():
async with client.messages.stream(
model="claude-opus-5",
max_tokens=4096,
messages=[{"role": "user", "content": body.question}],
) as stream:
async for text in stream.text_stream:
yield f"data: {json.dumps({'delta': text})}\n\n"
final = await stream.get_final_message()
yield f"data: {json.dumps({'done': True, 'usage': final.usage.model_dump()})}\n\n"
return StreamingResponse(events(), media_type="text/event-stream")
Server-sent events are the simplest transport that works: a plain HTTP response, one data: line
per chunk, and no WebSocket to manage. Send a final event carrying the usage so the client can
show what the answer cost.
Health and readiness
Two checks, and they are not the same:
A liveness check that calls the model API will restart your pods during an upstream incident. Keep
/healthz dependency-free and put the dependency checks in /readyz.
Running it
pip install "fastapi[standard]" uvicorn
uvicorn app:app --host 0.0.0.0 --port 8000 --workers 4
Because the handlers are async and almost all of their time is spent waiting on the model API,
a small number of workers serves a large number of concurrent requests — the module’s first lesson
applied to a server. Add the concurrency limit and the token bucket from lesson 2 inside the
process, so your own service shapes the load before the upstream API has to.
Try it yourself
Define AskRequest and AskResponse: a non-empty question of at most 500 characters, top_k between 1 and 20 defaulting to 5, and a response with an answer, a list of citations and a boolean refused.
Write status_for(error): map RateLimited to 429, Overloaded to 503, InvalidRequest to 400, NotFound to 404, and anything else to 500.
Check yourself
Why should /healthz not check the model API?
Check yourself
What does declaring a Pydantic response_model give you?
Recap
- Declare typed request and response models; validation, the 422 body and the OpenAPI schema all come from them.
- Inject the client with
Depends, and swap it in tests withdependency_overrides. - Map upstream failures onto honest status codes — a 429 with
Retry-After, not a 500. - Stream with
StreamingResponseand server-sent events, and send a final event carrying usage. /healthzmust be dependency-free; dependency checks belong in/readyz.- LLM endpoints are seconds-long and variable: long timeouts, streaming, and concurrency sized by your rate limit rather than by CPU.