Skip to content

Data Wrangling·Lesson 27 of 67·intermediate

Series and DataFrames

The two pandas objects everything else is built on — a labelled array and a labelled table — and the five commands you run on every dataset you meet.


In this lesson · 18 min

  • →Build Series and DataFrames from lists, dicts and records
  • →Read shape, columns, dtypes and summary statistics
  • →Understand what the index is and when to set or reset it
  • →Add and drop columns
  • →Round-trip a table through CSV and JSONL

pandas gives you a table with named columns, a labelled index, and a column-at-a-time way of working that keeps loops out of your data code. Most of an AI project is spent here: loading a dataset, finding out what is wrong with it, and reshaping it into what the model needs.

A Series

A Series is a 1-D array with an index attached:

PythonPython not started

The index does not have to be numbers, and that is the point — it lets you look things up by name:

PythonPython not started

A DataFrame

A DataFrame is a dict of Series that share one index — a table:

PythonPython not started

Real data usually arrives as a list of records — a JSON array, or the rows of a JSONL file — and pandas takes that directly:

PythonPython not started

The five commands you always run

Before anything else, find out what you have:

PythonPython not started
PythonPython not started

Columns

Selecting one column gives a Series; selecting a list of them gives a DataFrame:

PythonPython not started

New columns are computed from existing ones, a whole column at a time:

PythonPython not started

Note that no loop appeared. runs["tokens"] / (runs["latency_ms"] / 1000) is a vectorised operation over the whole column, exactly like NumPy.

The index

The index labels the rows. Setting a meaningful one makes lookups and joins natural:

PythonPython not started
PythonPython not started

Sorting

PythonPython not started

Reading and writing

PythonPython not started

Try it yourself

Exercise 1

Build the DataFrame runs from the records, add a cost_per_token column, and print the run name with the lowest cost per token.

Your turnPython not started
Exercise 2

Set top to the two highest-scoring rows, sorted best first, with the index renumbered from 0.

Your turnPython not started

Check yourself

What is the difference between df['score'] and df[['score']]?

Check yourself

You filter a DataFrame and the result's index starts at 3. Why, and when does it matter?

Recap

  • A Series is a labelled 1-D array; a DataFrame is a set of Series sharing one index.
  • pd.DataFrame(records) turns a list of dicts — the shape JSONL arrives in — into a table.
  • Always run head, shape, dtypes, describe and value_counts before anything else; info() shows where values are missing.
  • df["col"] is a Series, df[["col"]] is a DataFrame.
  • Build new columns from whole columns, never in a loop.
  • Filtering preserves the original index; reset_index(drop=True) renumbers when you want that.
  • to_csv(..., index=False) and to_json(..., orient="records", lines=True) round-trip cleanly.