Skip to content

Core Python·Lesson 12 of 67·intermediate

Iterators, generators and lazy pipelines

The protocol behind every for loop, how yield turns a function into a resumable stream, and how to chain generators into a pipeline that never loads the whole corpus.


In this lesson · 18 min

  • →Explain the iterator protocol — iter, next and StopIteration
  • →Write generator functions with yield, and understand where they pause
  • →Chain generators into a pipeline that processes one item at a time
  • →Delegate to a sub-generator with yield from
  • →Use itertools for the patterns you would otherwise write badly

Every for loop in Python is built on one small protocol. Understanding it is what lets you write a streaming pipeline instead of a program that loads ten gigabytes and dies.

The iterator protocol

iter() asks a collection for an iterator. next() pulls one item. When there are none left, next() raises StopIteration:

PythonPython not started

A for loop is exactly that, with the exception handled for you:

PythonPython not started

next() takes a default, which turns “nothing left” from an exception into a value — handy for “the first match, if there is one”:

PythonPython not started

That stops scanning the moment it finds one. Over a corpus of a million documents, that is the difference between milliseconds and minutes.

Generator functions

A function containing yield is a generator function. Calling it runs none of the body — it hands back a generator object. The body advances only when something asks for the next item, and it pauses again at each yield:

PythonPython not started

Watch where the prints appear relative to the next() calls: the body genuinely stops at yield and resumes there, keeping its local variables alive in between.

PythonPython not started

Streaming, not collecting

The point of a generator is that it never has to hold everything at once. Compare:

PythonPython not started

Here is what a streamed reply looks like, standing in for the real SDK:

PythonPython not started

Pipelines

Because a generator consumes an iterable and produces one, generators chain. Each stage pulls one item from the stage before it:

PythonPython not started

Nothing is computed until the for loop asks, and at any moment exactly one line is in flight. Add a stage, remove a stage, reorder them — the memory profile does not change.

yield from

To hand over every item of another iterable, yield from says it in one line:

PythonPython not started

itertools

The standard library already contains the pipeline stages you were about to write badly:

PythonPython not started

batched is the one you will reach for constantly: embedding and completion APIs take a limited number of inputs per request, and this is how you respect that limit.

PythonPython not started

Try it yourself

Exercise 1

Write the generator chunks(items, size), which yields lists of at most size items. Do not import itertools — build it with yield.

Your turnPython not started
Exercise 2

Print the id of the first document over 1000 tokens as first: ID, stopping the scan as soon as it is found. Print first: none if there is no such document.

Your turnPython not started

Check yourself

def g(): print('hi'); yield 1 — what happens when you call g() and assign it to a variable?

Check yourself

Why write tokenise(drop_blank(read(lines))) rather than three loops that build three lists?

Recap

  • iter() gives an iterator, next() pulls from it, StopIteration ends it — and for wraps all three. next(gen, default) turns the end into a value instead of an exception.
  • A function with yield is a generator function: calling it runs nothing, and the body pauses and resumes at each yield.
  • Generators chain into pipelines where exactly one item is in flight, whatever the input size.
  • yield from delegates to another iterable.
  • itertools already has islice, chain, batched, takewhile and count.
  • Generators are single-pass, and re-reading one fails silently rather than loudly.