Skip to content

Production AI Systems·Lesson 65 of 67·expert

Serving models with FastAPI

Typed request and response models, async endpoints, streaming, dependency injection for the client, and error handling that turns an upstream 429 into the right status code.


In this lesson · 22 min

  • →Define an endpoint with typed request and response models
  • →Validate input at the boundary and return a useful 422
  • →Inject the model client as a dependency so the app is testable
  • →Map upstream failures onto correct HTTP status codes
  • →Stream a response, and expose health and readiness checks

The smallest useful service

PythonPython not started

Everything typed in AskRequest is enforced before your function runs. Everything in AskResponse is enforced on the way out, so a handler cannot quietly return a shape the client does not expect.

Validation is free

PythonPython not started

Five cases, one model definition, no validation code. The 422 body names the field and the problem, which is what a client integrating against you needs.

Inject the client

The same seam as the testing lesson, expressed as a FastAPI dependency:

PythonPython not started

dependency_overrides is why this matters: the test swaps the client without touching the handler, and production wiring stays in one function.

Error handling

An upstream failure must become the right status code, not a 500:

PythonPython not started

Streaming

For a chat endpoint, streaming is the difference between a spinner and visible progress:

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/ask/stream")
async def ask_stream(body: AskRequest):
    async def events():
        async with client.messages.stream(
            model="claude-opus-5",
            max_tokens=4096,
            messages=[{"role": "user", "content": body.question}],
        ) as stream:
            async for text in stream.text_stream:
                yield f"data: {json.dumps({'delta': text})}\n\n"
            final = await stream.get_final_message()
            yield f"data: {json.dumps({'done': True, 'usage': final.usage.model_dump()})}\n\n"

    return StreamingResponse(events(), media_type="text/event-stream")

Server-sent events are the simplest transport that works: a plain HTTP response, one data: line per chunk, and no WebSocket to manage. Send a final event carrying the usage so the client can show what the answer cost.

Health and readiness

Two checks, and they are not the same:

PythonPython not started

A liveness check that calls the model API will restart your pods during an upstream incident. Keep /healthz dependency-free and put the dependency checks in /readyz.

Running it

pip install "fastapi[standard]" uvicorn
uvicorn app:app --host 0.0.0.0 --port 8000 --workers 4

Because the handlers are async and almost all of their time is spent waiting on the model API, a small number of workers serves a large number of concurrent requests — the module’s first lesson applied to a server. Add the concurrency limit and the token bucket from lesson 2 inside the process, so your own service shapes the load before the upstream API has to.

Try it yourself

Exercise 1

Define AskRequest and AskResponse: a non-empty question of at most 500 characters, top_k between 1 and 20 defaulting to 5, and a response with an answer, a list of citations and a boolean refused.

Your turnPython not started
Exercise 2

Write status_for(error): map RateLimited to 429, Overloaded to 503, InvalidRequest to 400, NotFound to 404, and anything else to 500.

Your turnPython not started

Check yourself

Why should /healthz not check the model API?

Check yourself

What does declaring a Pydantic response_model give you?

Recap

  • Declare typed request and response models; validation, the 422 body and the OpenAPI schema all come from them.
  • Inject the client with Depends, and swap it in tests with dependency_overrides.
  • Map upstream failures onto honest status codes — a 429 with Retry-After, not a 500.
  • Stream with StreamingResponse and server-sent events, and send a final event carrying usage.
  • /healthz must be dependency-free; dependency checks belong in /readyz.
  • LLM endpoints are seconds-long and variable: long timeouts, streaming, and concurrency sized by your rate limit rather than by CPU.