Skip to content

Core Python·Lesson 14 of 67·intermediate

Files, paths and JSON

pathlib instead of string surgery, context managers that always close, and JSON and JSONL — the two formats every dataset and API response arrives in.


In this lesson · 18 min

  • →Build and inspect paths with pathlib rather than string concatenation
  • →Read and write text safely with a with-block
  • →Stream a large file line by line instead of loading it whole
  • →Convert between Python objects and JSON, and handle what JSON cannot represent
  • →Read and write JSONL, the format datasets and eval sets ship in

Data arrives as files. In AI work that means a corpus of documents, a JSONL dataset, a directory of model outputs, and a config you have to read without breaking on a missing key.

Paths

pathlib treats a path as an object, not a string. The / operator joins, and it gets the separator right on every operating system:

PythonPython not started

Building paths by adding strings — "data" + "/" + name — breaks the first time a name already ends in a slash, and breaks again on Windows. Use /.

PythonPython not started

mkdir(parents=True, exist_ok=True) is the incantation worth memorising: create the whole chain, and do not complain if it is already there.

Reading and writing

For a small file, read_text and write_text are the whole API:

PythonPython not started

For anything bigger, open the file and let a with block close it for you — even if the code inside raises:

PythonPython not started

The modes you actually need: "r" read (the default), "w" write and truncate, "a" append, "rb"/"wb" for bytes.

PythonPython not started

Streaming a big file

Iterating the handle reads one line at a time, so the file’s size stops mattering:

PythonPython not started

JSON

json.dumps turns a Python object into a JSON string; json.loads turns it back:

PythonPython not started

Note what changed types on the way out and back: None became null, True became true, and a tuple would come back as a list. JSON has six types and Python has many more.

PythonPython not started

Reading and writing a JSON file uses dump and load — the same names without the s:

PythonPython not started

What JSON cannot hold

Sets, dates, bytes and NumPy numbers all fail. default= gives you a last-resort converter:

PythonPython not started
PythonPython not started

For text that is not English, turn off the escaping so the file stays readable:

PythonPython not started

JSONL

One JSON object per line. It is the format for datasets and eval sets, because you can append a record without rewriting the file and read a million of them without loading any but the current one.

PythonPython not started

A generator makes the reader reusable and keeps it lazy:

PythonPython not started

Try it yourself

Exercise 1

Write the three records to runs.jsonl, one JSON object per line, then read the file back and print total: N for the sum of their tokens.

Your turnPython not started
Exercise 2

json.dumps cannot serialise this record. Make it work by supplying a default= function: sets become sorted lists, and datetimes become ISO strings.

Your turnPython not started

Check yourself

Why is JSONL preferred over one big JSON array for a million-example dataset?

Check yourself

What does the with-block in `with open(path) as handle:` guarantee?

Recap

  • Build paths with pathlib and /, never string concatenation. mkdir(parents=True, exist_ok=True) creates the whole chain safely.
  • read_text/write_text for small files; with open(...) for everything else, always with encoding="utf-8".
  • Iterating a file handle yields one line at a time, so file size stops being a constraint.
  • json.dumps/loads work on strings, json.dump/load on file handles. Tuples come back as lists, and sets, dates and bytes need a default= converter.
  • JSONL — one JSON object per line — is how datasets, eval sets and logged outputs ship, because every line stands alone.