Skip to content

Production AI Systems·Lesson 59 of 67·expert

asyncio: concurrency for I/O-bound AI workloads

A hundred API calls one after another take a hundred times as long as one. Run them concurrently, bound the concurrency, and never block the event loop.


In this lesson · 22 min

  • →Explain when async helps and when it does nothing
  • →Write coroutines and run them concurrently with gather
  • →Bound concurrency with a semaphore
  • →Handle failures and timeouts without losing the other results
  • →Avoid blocking the event loop

When async helps

PythonPython not started

Sequential against concurrent

PythonPython not started

Ten calls that each wait 50 ms take 500 ms in sequence and about 50 ms together. Nothing got faster — the waiting simply overlapped.

PythonPython not started

Bounding concurrency

Launching a thousand calls at once will hit a rate limit, exhaust your connection pool, or both. A semaphore caps how many run at a time:

PythonPython not started

The semaphore is the single most important line in a production async client. Without it, concurrency is whatever the input size happens to be.

Failures

By default gather propagates the first exception and you lose every other result. return_exceptions=True keeps them:

PythonPython not started

For a batch job that is almost always what you want: finish everything, then report what failed.

PythonPython not started

Timeouts

A call that hangs holds a semaphore slot for ever. Bound it:

PythonPython not started

Results as they arrive

gather waits for everything. When you want to stream progress, take results as they finish:

PythonPython not started

Results arrive out of order, which is the point — a progress bar can move, and a downstream stage can start on the first result instead of waiting for the slowest.

Never block the loop

One synchronous call blocks everything, because there is only one thread:

PythonPython not started

In the first run every tick is stuck behind the blocking call. The rule: inside async def, never call anything that waits without await. If you must — a synchronous SDK, a CPU-heavy function — push it to a thread:

import asyncio

def parse_huge_file(path):          # synchronous, CPU-heavy
    ...

result = await asyncio.to_thread(parse_huge_file, "corpus.jsonl")

The shape in a real script

import asyncio, anthropic

client = anthropic.AsyncAnthropic()

async def answer(question: str, semaphore: asyncio.Semaphore) -> str:
    async with semaphore:
        async with asyncio.timeout(60):
            response = await client.messages.create(
                model="claude-opus-5",
                max_tokens=1024,
                messages=[{"role": "user", "content": question}],
            )
        return "".join(b.text for b in response.content if b.type == "text")

async def main() -> None:
    semaphore = asyncio.Semaphore(8)
    questions = load_questions()
    outcomes = await asyncio.gather(
        *(answer(q, semaphore) for q in questions),
        return_exceptions=True,
    )
    for question, outcome in zip(questions, outcomes):
        record(question, outcome)

asyncio.run(main())          # the entry point a script needs, and this page does not

Try it yourself

Exercise 1

Write run_limited(items, worker, limit): run worker over every item concurrently, with at most limit running at a time, returning the results in the original order.

Your turnPython not started
Exercise 2

Write settle(items, worker): run everything concurrently and return a list of (item, value, error) triples, so one failure never loses the other results.

Your turnPython not started

Check yourself

You replace ten sequential awaits with ten awaits inside a loop. Is the code now concurrent?

Check yourself

An async worker calls a synchronous library function that takes 200 ms. What happens?

Recap

  • async helps I/O-bound work — API calls, queries, fetches — and does nothing for CPU-bound work.
  • await alone is sequential. Concurrency comes from gather, TaskGroup or create_task.
  • Bound concurrency with asyncio.Semaphore, sized from your rate limit, not your input.
  • gather(..., return_exceptions=True) keeps every result when one call fails.
  • async with asyncio.timeout(n) stops a hung call holding a slot; as_completed streams results as they arrive.
  • One blocking call freezes the whole loop. Use asyncio.to_thread for synchronous work.