Skip to content

Production AI Systems·Lesson 67 of 67·expert

Cost, latency and capacity planning

Turn a product requirement into concurrency, a rate-limit headroom figure and a monthly bill — before you build it, and in numbers you can defend.


In this lesson · 22 min

  • →Report latency as percentiles, and separate time-to-first-token from total
  • →Size concurrency from throughput and latency with Little's law
  • →Build a cost model per request, per user and per month
  • →Check the plan against the rate limits you actually have
  • →Show what caching and model choice do to the total

The last lesson of the course is the one that decides whether a system ships. Everything before it makes an AI system work; this makes it affordable and fast enough to matter.

Latency is a distribution

PythonPython not started

A third of users are slower than the “average”, and the p99 is fifteen times the p50. Report percentiles; a mean of an AI workload describes nobody.

PythonPython not started

Two latencies, not one

PythonPython not started

Little’s law

The one piece of queueing theory worth memorising: concurrency = throughput × latency.

PythonPython not started

It runs backwards too, which is usually the more useful direction:

PythonPython not started

A cost model

PythonPython not started
PythonPython not started

Three changes, none of which alters the answer: cache the stable prefix, ask for a shorter reply, and stop sending context the model did not use. Together they take roughly two thirds off the bill.

Check it against the limits

PythonPython not started

The whole plan on one page

PythonPython not started

Three numbers change the shape of the whole plan: the peak multiplier (traffic is never flat), the cache hit rate (from module 6), and the p95 (which sets concurrency). Everything else follows.

Try it yourself

Exercise 1

Write concurrency(rate, latency) and max_rate(concurrency, latency) from Little's law, plus fits(rate, latency, limit) returning whether that rate is servable.

Your turnPython not started
Exercise 2

Write monthly_cost(...): the cost of a month of traffic, with a cache hit rate applied to the input tokens at one tenth the input rate.

Your turnPython not started

Check yourself

You must serve 12 requests per second, and the p95 latency is 4 seconds. How many requests are in flight at peak?

Check yourself

Your plan fits inside the published rate limit exactly, at 100%. Why is that not enough?

Recap

  • Report latency as percentiles and split time-to-first-token from total; the mean of an AI workload describes nobody.
  • Concurrency = throughput × latency. Size for the p95, and read it backwards to get the rate a concurrency limit allows.
  • A queue does not add capacity; it converts rejection into unbounded latency.
  • Build the cost model before building the system: caching, answer length and retrieved context are structural levers, and retrofitting them means re-running every evaluation.
  • Check requests per minute and tokens per minute against your limits, at about 70% of them.
  • Three numbers shape the plan: the peak multiplier, the cache hit rate and the p95.