Skip to content

Machine Learning Foundations·Lesson 37 of 67·advanced

Pipelines and preprocessing

Bind preprocessing to the model so it cannot leak, cannot be forgotten at prediction time, and travels as one object you can save.


In this lesson · 18 min

  • →Chain transformers and an estimator into a Pipeline
  • →Explain how a Pipeline prevents preprocessing leakage
  • →Handle numeric and categorical columns together with ColumnTransformer
  • →Impute missing values and encode categories safely
  • →Reach into a fitted pipeline's steps, and save the whole thing

The last lesson ended with a rule: fit transformers on the training data only. A Pipeline turns that rule into something the library enforces for you — and solves a second problem at the same time, because the preprocessing then travels with the model instead of living in a script somebody forgets to run.

A pipeline is an estimator

PythonPython not started

fit calls fit_transform on every step but the last, then fit on the last. predict calls transform on every step but the last, then predict. That asymmetry is the whole trick: the scaler learns from the training rows only, and the test rows are merely transformed.

PythonPython not started

make_pipeline names the steps for you from the class names, which is convenient when you do not need to refer to them.

Mixed column types

Real tables have numbers, categories and gaps, each needing different treatment. ColumnTransformer routes each group of columns to its own transformer:

PythonPython not started
PythonPython not started

Two numeric columns became two; two categorical columns with three values each became six. The pipeline knows how to redo all of it for a single new row.

PythonPython not started

Seeing what a step produced

PythonPython not started

set_output(transform="pandas") makes every step return a labelled DataFrame instead of a bare array, which turns debugging a preprocessing chain from guesswork into reading a table.

Saving the whole thing

import joblib

joblib.dump(model, "refusal-classifier.joblib")

# ...in the service, months later:
model = joblib.load("refusal-classifier.joblib")
prediction = model.predict(incoming_dataframe)

One file holds the imputer’s medians, the scaler’s statistics, the encoder’s categories and the classifier’s coefficients. Pin the scikit-learn version alongside it — a pickle is not portable across major versions.

Try it yourself

Exercise 1

Build model as a pipeline of a StandardScaler then a LogisticRegression, fit it on the training split, and put its held-out accuracy in accuracy.

Your turnPython not started
Exercise 2

Build preprocess: a ColumnTransformer that median-imputes then scales tokens and latency, and one-hot encodes model while ignoring unknown categories.

Your turnPython not started

Check yourself

How does a Pipeline prevent preprocessing leakage?

Check yourself

A request names a model released after training, and the service returns 500. What was missing?

Recap

  • A Pipeline is an estimator: fit, predict and score work on the whole chain.
  • It fits transformers on training rows only, which makes preprocessing leakage structurally impossible.
  • It keeps preprocessing bound to the model, so a service cannot forget a step.
  • ColumnTransformer routes numeric and categorical columns to different treatment; nest a Pipeline inside a branch to impute and then scale.
  • OneHotEncoder(handle_unknown="ignore") is required for anything that will meet new categories.
  • set_output(transform="pandas") makes intermediate output readable; joblib saves the whole fitted object.