Data Wrangling·Lesson 28 of 67·intermediate
Selecting, filtering and assigning
loc and iloc, boolean masks that read like sentences, and the chained-assignment mistake that silently does nothing.
In this lesson · 18 min
- →Select by label with loc and by position with iloc
- →Filter rows with boolean masks and combine conditions correctly
- →Use isin, between and the .str accessor for text columns
- →Assign through .loc rather than through a chained selection
- →Build derived columns with assign in a readable chain
Almost every question you ask of a dataset starts with “the rows where…”. pandas has two selectors and one filtering idiom, and knowing which is which saves a lot of confusion.
loc and iloc
loc works with labels — the index values and column names. iloc works with integer
positions, like a NumPy array:
Filtering
A comparison on a column gives a boolean Series, and passing it back selects the True rows:
Combine conditions with &, | and ~, parenthesising each one — exactly as in NumPy:
isin, between and .str
The .str accessor applies a string method down the whole column. .str.contains,
.str.lower, .str.strip, .str.replace and .str.len cover most text cleaning.
query
For a long chain of conditions, query reads better — column names become bare names:
An @ prefix reaches out to a Python variable. Note that query uses and/or, not &/| —
it is a small expression language of its own.
Assigning
Assign through .loc, naming both the rows and the column:
The mistake to know about is chained assignment — selecting twice and assigning into the second selection:
where, mask and assign
Try it yourself
Set slow_failures to the rows where the score is below 0.7 and latency is above 300 ms, keeping only the question and score columns.
The chained assignment below does nothing. Rewrite it with .loc so every score under 0.5 really does become 0.0.
Check yourself
df.loc['a':'c'] and df.iloc[0:3] on a six-row frame indexed a…f. How many rows does each return?
Check yourself
Why does df[df['score'] < 0.5]['score'] = 0.0 leave the original frame unchanged?
Recap
locselects by label and its slices include the endpoint;ilocselects by position and excludes it.- A comparison gives a boolean Series; index with it to filter, and combine with
&,|,~, parenthesising each condition. isin,betweenand the.straccessor cover most of what masks need.query("score > 0.7 and latency_ms < 400")reads better for long conditions, and@namereaches a Python variable.- Assign with
df.loc[rows, column] = value. Chained assignment writes to a temporary and is silently lost. assignreturns a new frame, so derived columns chain readably.