Skip to content

Core Python·Lesson 11 of 67·intermediate

Comprehensions and generator expressions

Build a list, dict or set in one expression — and learn when swapping the brackets for parentheses turns the whole thing lazy.


In this lesson · 16 min

  • →Write list, dict and set comprehensions with filters
  • →Flatten and nest comprehensions without losing the plot
  • →Swap a list comprehension for a generator expression when the data is large
  • →Pair generator expressions with sum, any, all, min and max
  • →Recognise when a plain for loop is the more readable choice

You met a list comprehension briefly in module 1. It is worth a lesson of its own, because comprehensions are how most idiomatic Python builds a collection — and because the lazy version is what lets you process a corpus that does not fit in memory.

PythonPython not started

The shape is always [ what to collect for name in iterable ].

Filtering

Add if at the end to keep only some items:

PythonPython not started

An if before the for is a different thing entirely — that is a conditional expression choosing what to collect, and it cannot skip items:

PythonPython not started

Dictionaries and sets

Change the brackets and you change the container:

PythonPython not started

Inverting a dictionary is a one-liner, and a useful reminder that keys must stay unique:

PythonPython not started

Nesting and flattening

Two for clauses read in the same order you would write the nested loop:

PythonPython not started

A comprehension whose expression is itself a comprehension builds a nested structure:

PythonPython not started

Generator expressions

Swap the square brackets for parentheses and nothing is built. You get a generator: an object that produces items one at a time, on demand.

PythonPython not started

That last line is the whole trade-off. A list can be read many times; a generator is a one-pass stream. What you get in return is memory:

PythonPython not started

Generators inside other functions

When a generator expression is the only argument, you can drop its parentheses. This is the form you will write most often:

PythonPython not started

any and all stop as soon as the answer is settled, so pairing them with a generator means you never do more work than the question requires:

PythonPython not started

When a loop is better

A comprehension should build one collection from one source. The moment it needs to do two things, keep running totals, or handle errors, write the loop:

PythonPython not started

Try it yourself

Exercise 1

Build long_docs: a dictionary of id → tokens holding only the documents with more than 500 tokens. Use a dict comprehension.

Your turnPython not started
Exercise 2

Print chars: N — the total number of characters across every chunk — without building an intermediate list.

Your turnPython not started

Check yourself

What is the difference between sum([len(c) for c in chunks]) and sum(len(c) for c in chunks)?

Check yourself

After g = (n for n in range(3)), what does list(g) return the second time you call it?

Recap

  • [expr for x in items if cond] builds a list; {} builds a dict or a set.
  • A trailing if filters; a leading if … else transforms and keeps every item.
  • Two for clauses read in the same order as the nested loop they replace. Stop at two.
  • Parentheses give a generator expression: lazy, one-pass, constant memory.
  • sum, any, all, min, max and join all take a generator directly, and any/all stop early.
  • If the body needs error handling or two outputs, write the loop instead.