Machine Learning Foundations·Lesson 37 of 67·advanced
Pipelines and preprocessing
Bind preprocessing to the model so it cannot leak, cannot be forgotten at prediction time, and travels as one object you can save.
In this lesson · 18 min
- →Chain transformers and an estimator into a Pipeline
- →Explain how a Pipeline prevents preprocessing leakage
- →Handle numeric and categorical columns together with ColumnTransformer
- →Impute missing values and encode categories safely
- →Reach into a fitted pipeline's steps, and save the whole thing
The last lesson ended with a rule: fit transformers on the training data only. A Pipeline
turns that rule into something the library enforces for you — and solves a second problem at the
same time, because the preprocessing then travels with the model instead of living in a script
somebody forgets to run.
A pipeline is an estimator
fit calls fit_transform on every step but the last, then fit on the last. predict calls
transform on every step but the last, then predict. That asymmetry is the whole trick: the
scaler learns from the training rows only, and the test rows are merely transformed.
make_pipeline names the steps for you from the class names, which is convenient when you do not
need to refer to them.
Mixed column types
Real tables have numbers, categories and gaps, each needing different treatment.
ColumnTransformer routes each group of columns to its own transformer:
Two numeric columns became two; two categorical columns with three values each became six. The pipeline knows how to redo all of it for a single new row.
Seeing what a step produced
set_output(transform="pandas") makes every step return a labelled DataFrame instead of a bare
array, which turns debugging a preprocessing chain from guesswork into reading a table.
Saving the whole thing
import joblib
joblib.dump(model, "refusal-classifier.joblib")
# ...in the service, months later:
model = joblib.load("refusal-classifier.joblib")
prediction = model.predict(incoming_dataframe)
One file holds the imputer’s medians, the scaler’s statistics, the encoder’s categories and the classifier’s coefficients. Pin the scikit-learn version alongside it — a pickle is not portable across major versions.
Try it yourself
Build model as a pipeline of a StandardScaler then a LogisticRegression, fit it on the training split, and put its held-out accuracy in accuracy.
Build preprocess: a ColumnTransformer that median-imputes then scales tokens and latency, and one-hot encodes model while ignoring unknown categories.
Check yourself
How does a Pipeline prevent preprocessing leakage?
Check yourself
A request names a model released after training, and the service returns 500. What was missing?
Recap
- A
Pipelineis an estimator:fit,predictandscorework on the whole chain. - It fits transformers on training rows only, which makes preprocessing leakage structurally impossible.
- It keeps preprocessing bound to the model, so a service cannot forget a step.
ColumnTransformerroutes numeric and categorical columns to different treatment; nest aPipelineinside a branch to impute and then scale.OneHotEncoder(handle_unknown="ignore")is required for anything that will meet new categories.set_output(transform="pandas")makes intermediate output readable;joblibsaves the whole fitted object.