Skip to content

LLM Engineering·Lesson 51 of 67·advanced

Token accounting, caching and cost control

Count tokens properly, cache the stable prefix, find the one-character change that silently destroys your hit rate, and know which levers are free before you trade quality for money.


In this lesson · 22 min

  • →Count tokens with the API rather than guessing
  • →Read every field of the usage block, including the cache ones
  • →Structure a request so its prefix is cacheable
  • →Find the silent invalidators that take a hit rate to zero
  • →Order the cost levers: free wins before quality trades

Everything in this module has a price, and almost all of it is input tokens you sent more than once. This lesson is about measuring that and then reducing it — in the order that costs you nothing first.

Counting tokens

Ask the API. Token counts are model-specific, and the endpoint is free:

import anthropic

client = anthropic.Anthropic()

count = client.messages.count_tokens(
    model="claude-opus-5",
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": document}],
)
print(count.input_tokens)

A rough local estimate is fine for a sanity check, as long as you know it is rough:

PythonPython not started

The usage block

PythonPython not started

How caching works

The cache is a prefix match. The request is rendered in a fixed order, and a breakpoint says “everything up to here is cacheable”:

tools  ->  system  ->  messages

Any byte that changes anywhere in that prefix invalidates the cache from that point on. So the architecture is simple to state and easy to get wrong: stable content first, volatile content last.

PythonPython not started

On your own machine, the simplest form caches the last cacheable block automatically:

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    cache_control={"type": "ephemeral"},        # cache the last cacheable block
    system=LARGE_STABLE_SYSTEM_PROMPT,
    messages=[{"role": "user", "content": question}],
)

Or place breakpoints yourself, when you want the boundary in a specific spot:

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    system=[
        {"type": "text", "text": STABLE_INSTRUCTIONS},
        {"type": "text", "text": THE_WHOLE_HANDBOOK,
         "cache_control": {"type": "ephemeral", "ttl": "1h"}},
    ],
    messages=[{"role": "user", "content": question}],
)

At most four breakpoints per request. The default time to live is five minutes, refreshed on each hit; "1h" is available when your traffic is spikier than that.

Silent invalidators

This is where hit rates actually go to die:

PythonPython not started
PythonPython not started

Two identical tool sets, two different cache keys — unless you sort them. Verify in production rather than assuming:

PythonPython not started

That second pattern — writing the cache on every single call and never reading it — costs you more than not caching at all, because writes are 1.25x.

The order of the levers

PythonPython not started

The batch path is worth knowing, because half price for no quality change is unusual:

batch = client.messages.batches.create(requests=[
    {"custom_id": "q-1", "params": {"model": "claude-opus-5", "max_tokens": 1024,
                                    "messages": [{"role": "user", "content": "..."}]}},
    {"custom_id": "q-2", "params": {...}},
])

while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(30)

for entry in client.messages.batches.results(batch.id):
    if entry.result.type == "succeeded":
        store(entry.custom_id, entry.result.message)

Try it yourself

Exercise 1

Write break_even(prefix_tokens): how many requests before caching a prefix of that size is cheaper than not caching, given a write of 1.25x and a read of 0.1x.

Your turnPython not started
Exercise 2

Write first_difference(a, b): the index of the first character where two rendered prefixes diverge, or None when they are identical — the fastest way to find a cache invalidator.

Your turnPython not started

Check yourself

Your cache_read_input_tokens is zero across thousands of requests with the same system prompt. What is happening?

Check yourself

Which cost lever should you exhaust before trading any quality?

Recap

  • Count tokens with client.messages.count_tokens, never with tiktoken.
  • usage has four fields: uncached input, output, cache writes (~1.25x) and cache reads (~0.1x).
  • The cache is a prefix match over tools -> system -> messages. Stable content first, volatile content after the last breakpoint. At most four breakpoints, and a minimum prefix size below which nothing caches silently.
  • Timestamps, UUIDs, unsorted JSON and reordered tool lists are the invalidators to hunt for; a hit rate near zero with repeated writes costs more than no cache at all.
  • Do the free levers — caching, input and output hygiene, batch — before trading any quality, and judge cost per completed task.