Production AI Systems·Lesson 67 of 67·expert
Cost, latency and capacity planning
Turn a product requirement into concurrency, a rate-limit headroom figure and a monthly bill — before you build it, and in numbers you can defend.
In this lesson · 22 min
- →Report latency as percentiles, and separate time-to-first-token from total
- →Size concurrency from throughput and latency with Little's law
- →Build a cost model per request, per user and per month
- →Check the plan against the rate limits you actually have
- →Show what caching and model choice do to the total
The last lesson of the course is the one that decides whether a system ships. Everything before it makes an AI system work; this makes it affordable and fast enough to matter.
Latency is a distribution
A third of users are slower than the “average”, and the p99 is fifteen times the p50. Report percentiles; a mean of an AI workload describes nobody.
Two latencies, not one
Little’s law
The one piece of queueing theory worth memorising: concurrency = throughput × latency.
It runs backwards too, which is usually the more useful direction:
A cost model
Three changes, none of which alters the answer: cache the stable prefix, ask for a shorter reply, and stop sending context the model did not use. Together they take roughly two thirds off the bill.
Check it against the limits
The whole plan on one page
Three numbers change the shape of the whole plan: the peak multiplier (traffic is never flat), the cache hit rate (from module 6), and the p95 (which sets concurrency). Everything else follows.
Try it yourself
Write concurrency(rate, latency) and max_rate(concurrency, latency) from Little's law, plus fits(rate, latency, limit) returning whether that rate is servable.
Write monthly_cost(...): the cost of a month of traffic, with a cache hit rate applied to the input tokens at one tenth the input rate.
Check yourself
You must serve 12 requests per second, and the p95 latency is 4 seconds. How many requests are in flight at peak?
Check yourself
Your plan fits inside the published rate limit exactly, at 100%. Why is that not enough?
Recap
- Report latency as percentiles and split time-to-first-token from total; the mean of an AI workload describes nobody.
- Concurrency = throughput × latency. Size for the p95, and read it backwards to get the rate a concurrency limit allows.
- A queue does not add capacity; it converts rejection into unbounded latency.
- Build the cost model before building the system: caching, answer length and retrieved context are structural levers, and retrofitting them means re-running every evaluation.
- Check requests per minute and tokens per minute against your limits, at about 70% of them.
- Three numbers shape the plan: the peak multiplier, the cache hit rate and the p95.