Core Python·Lesson 19 of 67·intermediate
The standard library tour for AI work
The dozen modules that turn up in every AI codebase — collections, statistics, random and seeding, hashlib for cache keys, re for cleaning text, and os.environ for the API key you must never hard-code.
In this lesson · 20 min
- →Reach for Counter, defaultdict and deque instead of reinventing them
- →Summarise numbers with statistics, and seed random for reproducible runs
- →Hash content to build stable cache and de-duplication keys
- →Clean and extract text with regular expressions
- →Read configuration and secrets from the environment
Python ships with most of what an AI pipeline needs before you install anything. This is a tour of the parts you will actually use, and why each one turns up.
collections
Counter counts, defaultdict groups, deque is a queue with a fixed memory:
A deque(maxlen=n) is the sliding context window in one line: append forever, and it silently
drops the oldest.
statistics
The mean is 387 ms and the median is 115 ms. Report the median and a high percentile for latency — one slow request should not be allowed to describe the typical experience.
random, and seeding it
hashlib
A hash turns content into a short, stable fingerprint. It is how you build a cache key, spot a duplicate document, and give a chunk an id that does not change between runs:
Regular expressions
re is for finding and reshaping patterns in text. Always write the pattern as a raw string, so
backslashes reach the regex engine intact:
datetime
Store timestamps in UTC and as ISO strings. A naive datetime.now() is ambiguous the moment your
code runs on a machine in another timezone, which it will.
Timing
Use time.perf_counter() for durations — it is monotonic, so it cannot go backwards when the
system clock is adjusted. time.time() is for wall-clock timestamps, not measurement.
Environment variables
Configuration and secrets come from the environment, never from the source file:
Note that every value arrives as a string: int(...) for numbers, and an explicit comparison
for booleans, because bool("false") is True.
textwrap and uuid
textwrap.dedent is the one to remember for prompts written as triple-quoted strings — it strips
the indentation your code has but your prompt should not:
Try it yourself
Group the documents by language into by_language (a plain dict of language → list of ids), and build totals: a dict of language → total tokens.
Write cache_key(model, prompt, **options): a 16-character SHA-256 hex digest that is stable across runs and independent of the order the options are passed in.
Check yourself
Why use hashlib.sha256 rather than the built-in hash() for a cache key you write to disk?
Check yourself
You report a model's accuracy after shuffling the dataset without setting a seed. What is wrong?
Recap
Countercounts,defaultdictgroups without key checks, anddeque(maxlen=n)is a sliding window.- Report the median and a percentile for latency, not the mean.
- Seed every source of randomness and record the seed;
random.Random(seed)keeps it private. hashlibgives stable fingerprints for cache keys and de-duplication; the built-inhash()does not survive a restart.- Use raw strings for regex patterns, and a string method when one will do.
- Keep timestamps timezone-aware and in UTC; measure durations with
time.perf_counter(). - Read configuration and secrets from
os.environ, converting the strings it hands you.