Skip to content

Data Wrangling·Lesson 28 of 67·intermediate

Selecting, filtering and assigning

loc and iloc, boolean masks that read like sentences, and the chained-assignment mistake that silently does nothing.


In this lesson · 18 min

  • →Select by label with loc and by position with iloc
  • →Filter rows with boolean masks and combine conditions correctly
  • →Use isin, between and the .str accessor for text columns
  • →Assign through .loc rather than through a chained selection
  • →Build derived columns with assign in a readable chain

Almost every question you ask of a dataset starts with “the rows where…”. pandas has two selectors and one filtering idiom, and knowing which is which saves a lot of confusion.

PythonPython not started

loc and iloc

loc works with labels — the index values and column names. iloc works with integer positions, like a NumPy array:

PythonPython not started

Filtering

A comparison on a column gives a boolean Series, and passing it back selects the True rows:

PythonPython not started

Combine conditions with &, | and ~, parenthesising each one — exactly as in NumPy:

PythonPython not started

isin, between and .str

PythonPython not started

The .str accessor applies a string method down the whole column. .str.contains, .str.lower, .str.strip, .str.replace and .str.len cover most text cleaning.

PythonPython not started

query

For a long chain of conditions, query reads better — column names become bare names:

PythonPython not started

An @ prefix reaches out to a Python variable. Note that query uses and/or, not &/| — it is a small expression language of its own.

Assigning

Assign through .loc, naming both the rows and the column:

PythonPython not started

The mistake to know about is chained assignment — selecting twice and assigning into the second selection:

PythonPython not started

where, mask and assign

PythonPython not started

Try it yourself

Exercise 1

Set slow_failures to the rows where the score is below 0.7 and latency is above 300 ms, keeping only the question and score columns.

Your turnPython not started
Exercise 2

The chained assignment below does nothing. Rewrite it with .loc so every score under 0.5 really does become 0.0.

Your turnPython not started

Check yourself

df.loc['a':'c'] and df.iloc[0:3] on a six-row frame indexed a…f. How many rows does each return?

Check yourself

Why does df[df['score'] < 0.5]['score'] = 0.0 leave the original frame unchanged?

Recap

  • loc selects by label and its slices include the endpoint; iloc selects by position and excludes it.
  • A comparison gives a boolean Series; index with it to filter, and combine with &, |, ~, parenthesising each condition.
  • isin, between and the .str accessor cover most of what masks need.
  • query("score > 0.7 and latency_ms < 400") reads better for long conditions, and @name reaches a Python variable.
  • Assign with df.loc[rows, column] = value. Chained assignment writes to a temporary and is silently lost.
  • assign returns a new frame, so derived columns chain readably.