Production AI Systems·Lesson 59 of 67·expert
asyncio: concurrency for I/O-bound AI workloads
A hundred API calls one after another take a hundred times as long as one. Run them concurrently, bound the concurrency, and never block the event loop.
In this lesson · 22 min
- →Explain when async helps and when it does nothing
- →Write coroutines and run them concurrently with gather
- →Bound concurrency with a semaphore
- →Handle failures and timeouts without losing the other results
- →Avoid blocking the event loop
When async helps
Sequential against concurrent
Ten calls that each wait 50 ms take 500 ms in sequence and about 50 ms together. Nothing got faster — the waiting simply overlapped.
Bounding concurrency
Launching a thousand calls at once will hit a rate limit, exhaust your connection pool, or both. A semaphore caps how many run at a time:
The semaphore is the single most important line in a production async client. Without it, concurrency is whatever the input size happens to be.
Failures
By default gather propagates the first exception and you lose every other result.
return_exceptions=True keeps them:
For a batch job that is almost always what you want: finish everything, then report what failed.
Timeouts
A call that hangs holds a semaphore slot for ever. Bound it:
Results as they arrive
gather waits for everything. When you want to stream progress, take results as they finish:
Results arrive out of order, which is the point — a progress bar can move, and a downstream stage can start on the first result instead of waiting for the slowest.
Never block the loop
One synchronous call blocks everything, because there is only one thread:
In the first run every tick is stuck behind the blocking call. The rule: inside async def, never
call anything that waits without await. If you must — a synchronous SDK, a CPU-heavy function —
push it to a thread:
import asyncio
def parse_huge_file(path): # synchronous, CPU-heavy
...
result = await asyncio.to_thread(parse_huge_file, "corpus.jsonl")
The shape in a real script
import asyncio, anthropic
client = anthropic.AsyncAnthropic()
async def answer(question: str, semaphore: asyncio.Semaphore) -> str:
async with semaphore:
async with asyncio.timeout(60):
response = await client.messages.create(
model="claude-opus-5",
max_tokens=1024,
messages=[{"role": "user", "content": question}],
)
return "".join(b.text for b in response.content if b.type == "text")
async def main() -> None:
semaphore = asyncio.Semaphore(8)
questions = load_questions()
outcomes = await asyncio.gather(
*(answer(q, semaphore) for q in questions),
return_exceptions=True,
)
for question, outcome in zip(questions, outcomes):
record(question, outcome)
asyncio.run(main()) # the entry point a script needs, and this page does not
Try it yourself
Write run_limited(items, worker, limit): run worker over every item concurrently, with at most limit running at a time, returning the results in the original order.
Write settle(items, worker): run everything concurrently and return a list of (item, value, error) triples, so one failure never loses the other results.
Check yourself
You replace ten sequential awaits with ten awaits inside a loop. Is the code now concurrent?
Check yourself
An async worker calls a synchronous library function that takes 200 ms. What happens?
Recap
- async helps I/O-bound work — API calls, queries, fetches — and does nothing for CPU-bound work.
awaitalone is sequential. Concurrency comes fromgather,TaskGrouporcreate_task.- Bound concurrency with
asyncio.Semaphore, sized from your rate limit, not your input. gather(..., return_exceptions=True)keeps every result when one call fails.async with asyncio.timeout(n)stops a hung call holding a slot;as_completedstreams results as they arrive.- One blocking call freezes the whole loop. Use
asyncio.to_threadfor synchronous work.