Skip to content

Data Wrangling·Lesson 29 of 67·intermediate

Missing data and type coercion

Find what is missing, decide honestly what to do about it, and fix the column that arrived as text — the two problems that quietly poison a model downstream.


In this lesson · 18 min

  • →Find missing values and quantify them per column
  • →Choose between dropping, filling and leaving values missing
  • →Fill forward and backward, and know when that is wrong
  • →Coerce a text column into numbers or dates without losing the bad rows silently
  • →Use categorical and nullable-integer dtypes deliberately
  • →Find and remove duplicate rows

Real datasets are incomplete and badly typed. Neither problem announces itself: a model trains happily on a column of strings that should have been numbers, and a mean-filled column looks perfectly reasonable right up until you notice the test scores were leaked into it.

Finding what is missing

PythonPython not started

Dropping

PythonPython not started

dropna() with no arguments removes a row if anything in it is missing, which on a wide table can delete most of your data. Almost always you want subset= — the columns that actually matter for what you are about to do.

Filling

PythonPython not started

For ordered data — a time series, a log — carrying the last known value forward is often the honest choice:

PythonPython not started

Type coercion

A single stray value turns a whole numeric column into text, and arithmetic then either fails or does something absurd:

PythonPython not started

pd.to_numeric with errors="coerce" converts what it can and marks the rest as missing, so the bad rows become visible instead of invisible:

PythonPython not started

The same applies to dates:

PythonPython not started

Dtypes that save memory and mistakes

A column with few distinct values should be category; a column of counts with gaps should be a nullable integer:

PythonPython not started

Duplicates

PythonPython not started

Try it yourself

Exercise 1

Coerce tokens to numbers, put the failures' ids in failed, and fill the resulting gaps with the column's median into a new tokens_filled column.

Your turnPython not started
Exercise 2

Remove documents whose text has already been seen, keeping the first occurrence, into unique_docs. Then set removed to how many rows went.

Your turnPython not started

Check yourself

You fill a column's missing values with its mean, then split into train and test. What is wrong?

Check yourself

Why does pd.to_numeric(col, errors='coerce') beat quietly skipping the bad rows?

Recap

  • df.isna().sum() counts gaps; df.isna().mean() gives the proportion, which is what to judge.
  • dropna() with no subset deletes a row for any gap anywhere — nearly always pass subset=.
  • fillna takes a constant or a per-column dict; ffill carries the last value forward, bfill carries the future backwards and leaks it.
  • Compute imputation statistics on the training set only.
  • pd.to_numeric(..., errors="coerce") and pd.to_datetime(..., errors="coerce") make bad rows visible as NaN instead of losing them.
  • category shrinks a low-cardinality text column; Int64 keeps integers that have gaps.
  • De-duplicate on content before splitting, or your evaluation measures memorisation.