Data Wrangling·Lesson 27 of 67·intermediate
Series and DataFrames
The two pandas objects everything else is built on — a labelled array and a labelled table — and the five commands you run on every dataset you meet.
In this lesson · 18 min
- →Build Series and DataFrames from lists, dicts and records
- →Read shape, columns, dtypes and summary statistics
- →Understand what the index is and when to set or reset it
- →Add and drop columns
- →Round-trip a table through CSV and JSONL
pandas gives you a table with named columns, a labelled index, and a column-at-a-time way of working that keeps loops out of your data code. Most of an AI project is spent here: loading a dataset, finding out what is wrong with it, and reshaping it into what the model needs.
A Series
A Series is a 1-D array with an index attached:
The index does not have to be numbers, and that is the point — it lets you look things up by name:
A DataFrame
A DataFrame is a dict of Series that share one index — a table:
Real data usually arrives as a list of records — a JSON array, or the rows of a JSONL file — and pandas takes that directly:
The five commands you always run
Before anything else, find out what you have:
Columns
Selecting one column gives a Series; selecting a list of them gives a DataFrame:
New columns are computed from existing ones, a whole column at a time:
Note that no loop appeared. runs["tokens"] / (runs["latency_ms"] / 1000) is a vectorised
operation over the whole column, exactly like NumPy.
The index
The index labels the rows. Setting a meaningful one makes lookups and joins natural:
Sorting
Reading and writing
Try it yourself
Build the DataFrame runs from the records, add a cost_per_token column, and print the run name with the lowest cost per token.
Set top to the two highest-scoring rows, sorted best first, with the index renumbered from 0.
Check yourself
What is the difference between df['score'] and df[['score']]?
Check yourself
You filter a DataFrame and the result's index starts at 3. Why, and when does it matter?
Recap
- A Series is a labelled 1-D array; a DataFrame is a set of Series sharing one index.
pd.DataFrame(records)turns a list of dicts — the shape JSONL arrives in — into a table.- Always run
head,shape,dtypes,describeandvalue_countsbefore anything else;info()shows where values are missing. df["col"]is a Series,df[["col"]]is a DataFrame.- Build new columns from whole columns, never in a loop.
- Filtering preserves the original index;
reset_index(drop=True)renumbers when you want that. to_csv(..., index=False)andto_json(..., orient="records", lines=True)round-trip cleanly.