LLM Engineering·Lesson 51 of 67·advanced
Token accounting, caching and cost control
Count tokens properly, cache the stable prefix, find the one-character change that silently destroys your hit rate, and know which levers are free before you trade quality for money.
In this lesson · 22 min
- →Count tokens with the API rather than guessing
- →Read every field of the usage block, including the cache ones
- →Structure a request so its prefix is cacheable
- →Find the silent invalidators that take a hit rate to zero
- →Order the cost levers: free wins before quality trades
Everything in this module has a price, and almost all of it is input tokens you sent more than once. This lesson is about measuring that and then reducing it — in the order that costs you nothing first.
Counting tokens
Ask the API. Token counts are model-specific, and the endpoint is free:
import anthropic
client = anthropic.Anthropic()
count = client.messages.count_tokens(
model="claude-opus-5",
system=SYSTEM_PROMPT,
messages=[{"role": "user", "content": document}],
)
print(count.input_tokens)
A rough local estimate is fine for a sanity check, as long as you know it is rough:
The usage block
How caching works
The cache is a prefix match. The request is rendered in a fixed order, and a breakpoint says “everything up to here is cacheable”:
tools -> system -> messages
Any byte that changes anywhere in that prefix invalidates the cache from that point on. So the architecture is simple to state and easy to get wrong: stable content first, volatile content last.
On your own machine, the simplest form caches the last cacheable block automatically:
response = client.messages.create(
model="claude-opus-5",
max_tokens=16000,
cache_control={"type": "ephemeral"}, # cache the last cacheable block
system=LARGE_STABLE_SYSTEM_PROMPT,
messages=[{"role": "user", "content": question}],
)
Or place breakpoints yourself, when you want the boundary in a specific spot:
response = client.messages.create(
model="claude-opus-5",
max_tokens=16000,
system=[
{"type": "text", "text": STABLE_INSTRUCTIONS},
{"type": "text", "text": THE_WHOLE_HANDBOOK,
"cache_control": {"type": "ephemeral", "ttl": "1h"}},
],
messages=[{"role": "user", "content": question}],
)
At most four breakpoints per request. The default time to live is five minutes, refreshed on each
hit; "1h" is available when your traffic is spikier than that.
Silent invalidators
This is where hit rates actually go to die:
Two identical tool sets, two different cache keys — unless you sort them. Verify in production rather than assuming:
That second pattern — writing the cache on every single call and never reading it — costs you more than not caching at all, because writes are 1.25x.
The order of the levers
The batch path is worth knowing, because half price for no quality change is unusual:
batch = client.messages.batches.create(requests=[
{"custom_id": "q-1", "params": {"model": "claude-opus-5", "max_tokens": 1024,
"messages": [{"role": "user", "content": "..."}]}},
{"custom_id": "q-2", "params": {...}},
])
while client.messages.batches.retrieve(batch.id).processing_status != "ended":
time.sleep(30)
for entry in client.messages.batches.results(batch.id):
if entry.result.type == "succeeded":
store(entry.custom_id, entry.result.message)
Try it yourself
Write break_even(prefix_tokens): how many requests before caching a prefix of that size is cheaper than not caching, given a write of 1.25x and a read of 0.1x.
Write first_difference(a, b): the index of the first character where two rendered prefixes diverge, or None when they are identical — the fastest way to find a cache invalidator.
Check yourself
Your cache_read_input_tokens is zero across thousands of requests with the same system prompt. What is happening?
Check yourself
Which cost lever should you exhaust before trading any quality?
Recap
- Count tokens with
client.messages.count_tokens, never withtiktoken. usagehas four fields: uncached input, output, cache writes (~1.25x) and cache reads (~0.1x).- The cache is a prefix match over
tools -> system -> messages. Stable content first, volatile content after the last breakpoint. At most four breakpoints, and a minimum prefix size below which nothing caches silently. - Timestamps, UUIDs, unsorted JSON and reordered tool lists are the invalidators to hunt for; a hit rate near zero with repeated writes costs more than no cache at all.
- Do the free levers — caching, input and output hygiene, batch — before trading any quality, and judge cost per completed task.