Core Python·Lesson 12 of 67·intermediate
Iterators, generators and lazy pipelines
The protocol behind every for loop, how yield turns a function into a resumable stream, and how to chain generators into a pipeline that never loads the whole corpus.
In this lesson · 18 min
- →Explain the iterator protocol — iter, next and StopIteration
- →Write generator functions with yield, and understand where they pause
- →Chain generators into a pipeline that processes one item at a time
- →Delegate to a sub-generator with yield from
- →Use itertools for the patterns you would otherwise write badly
Every for loop in Python is built on one small protocol. Understanding it is what lets you
write a streaming pipeline instead of a program that loads ten gigabytes and dies.
The iterator protocol
iter() asks a collection for an iterator. next() pulls one item. When there are none left,
next() raises StopIteration:
A for loop is exactly that, with the exception handled for you:
next() takes a default, which turns “nothing left” from an exception into a value — handy for
“the first match, if there is one”:
That stops scanning the moment it finds one. Over a corpus of a million documents, that is the difference between milliseconds and minutes.
Generator functions
A function containing yield is a generator function. Calling it runs none of the body — it
hands back a generator object. The body advances only when something asks for the next item, and
it pauses again at each yield:
Watch where the prints appear relative to the next() calls: the body genuinely stops at
yield and resumes there, keeping its local variables alive in between.
Streaming, not collecting
The point of a generator is that it never has to hold everything at once. Compare:
Here is what a streamed reply looks like, standing in for the real SDK:
Pipelines
Because a generator consumes an iterable and produces one, generators chain. Each stage pulls one item from the stage before it:
Nothing is computed until the for loop asks, and at any moment exactly one line is in flight.
Add a stage, remove a stage, reorder them — the memory profile does not change.
yield from
To hand over every item of another iterable, yield from says it in one line:
itertools
The standard library already contains the pipeline stages you were about to write badly:
batched is the one you will reach for constantly: embedding and completion APIs take a limited
number of inputs per request, and this is how you respect that limit.
Try it yourself
Write the generator chunks(items, size), which yields lists of at most size items. Do not import itertools — build it with yield.
Print the id of the first document over 1000 tokens as first: ID, stopping the scan as soon as it is found. Print first: none if there is no such document.
Check yourself
def g(): print('hi'); yield 1 — what happens when you call g() and assign it to a variable?
Check yourself
Why write tokenise(drop_blank(read(lines))) rather than three loops that build three lists?
Recap
iter()gives an iterator,next()pulls from it,StopIterationends it — andforwraps all three.next(gen, default)turns the end into a value instead of an exception.- A function with
yieldis a generator function: calling it runs nothing, and the body pauses and resumes at eachyield. - Generators chain into pipelines where exactly one item is in flight, whatever the input size.
yield fromdelegates to another iterable.itertoolsalready hasislice,chain,batched,takewhileandcount.- Generators are single-pass, and re-reading one fails silently rather than loudly.