Data Wrangling·Lesson 29 of 67·intermediate
Missing data and type coercion
Find what is missing, decide honestly what to do about it, and fix the column that arrived as text — the two problems that quietly poison a model downstream.
In this lesson · 18 min
- →Find missing values and quantify them per column
- →Choose between dropping, filling and leaving values missing
- →Fill forward and backward, and know when that is wrong
- →Coerce a text column into numbers or dates without losing the bad rows silently
- →Use categorical and nullable-integer dtypes deliberately
- →Find and remove duplicate rows
Real datasets are incomplete and badly typed. Neither problem announces itself: a model trains happily on a column of strings that should have been numbers, and a mean-filled column looks perfectly reasonable right up until you notice the test scores were leaked into it.
Finding what is missing
Dropping
dropna() with no arguments removes a row if anything in it is missing, which on a wide table
can delete most of your data. Almost always you want subset= — the columns that actually
matter for what you are about to do.
Filling
For ordered data — a time series, a log — carrying the last known value forward is often the honest choice:
Type coercion
A single stray value turns a whole numeric column into text, and arithmetic then either fails or does something absurd:
pd.to_numeric with errors="coerce" converts what it can and marks the rest as missing, so the
bad rows become visible instead of invisible:
The same applies to dates:
Dtypes that save memory and mistakes
A column with few distinct values should be category; a column of counts with gaps should be a
nullable integer:
Duplicates
Try it yourself
Coerce tokens to numbers, put the failures' ids in failed, and fill the resulting gaps with the column's median into a new tokens_filled column.
Remove documents whose text has already been seen, keeping the first occurrence, into unique_docs. Then set removed to how many rows went.
Check yourself
You fill a column's missing values with its mean, then split into train and test. What is wrong?
Check yourself
Why does pd.to_numeric(col, errors='coerce') beat quietly skipping the bad rows?
Recap
df.isna().sum()counts gaps;df.isna().mean()gives the proportion, which is what to judge.dropna()with nosubsetdeletes a row for any gap anywhere — nearly always passsubset=.fillnatakes a constant or a per-column dict;ffillcarries the last value forward,bfillcarries the future backwards and leaks it.- Compute imputation statistics on the training set only.
pd.to_numeric(..., errors="coerce")andpd.to_datetime(..., errors="coerce")make bad rows visible asNaNinstead of losing them.categoryshrinks a low-cardinality text column;Int64keeps integers that have gaps.- De-duplicate on content before splitting, or your evaluation measures memorisation.