Feature Engineering for Machine Learning
Feature engineering is the process of using domain knowledge and data transformation techniques to create, select, or modify input variables (features) that improve machine learning model performance. It is often the single most impactful step in the ML pipeline — transforming raw data into a representation that algorithms can learn from effectively.
“Feature engineering is the art of turning raw data into features that make machine learning models work.” — Pedro Domingos, The Master Algorithm
Why Feature Engineering Matters
Raw data is rarely in the form best suited for a learning algorithm. Consider:
- A date column — is it more useful as a raw timestamp, or as day-of-week + hour + is_weekend?
- A categorical “city” feature with 1,000 values — one-hot encoding creates 1,000 columns; target encoding or embeddings may be far more compact and informative.
- Two separately weak features — their ratio or interaction might be highly predictive.
Feature engineering bridges the gap between domain understanding and algorithmic learning.
Numerical Feature Transformations
Scaling
Most algorithms that use distance metrics or gradient descent are sensitive to feature scale:
| Method | Formula | Use When |
|---|---|---|
| Min-Max Scaling | $\frac{x - x_{min}}{x_{max} - x_{min}}$ | Bounded range needed; no outliers |
| Standardization (Z-score) | $\frac{x - \mu}{\sigma}$ | Normal-ish distribution; outliers present |
| Robust Scaling | $\frac{x - Q_2}{Q_3 - Q_1}$ | Significant outliers |
| Log Transform | $\log(x + 1)$ | Right-skewed distribution |
| Box-Cox | Parametric power transform | Normalize any continuous feature |
Binning / Discretization
Continuous values are grouped into bins — converting a regression-style feature into a categorical one. Useful when relationships are non-linear or when domain expertise defines meaningful ranges (age groups, income brackets).
Polynomial Features
Creating interaction and higher-order terms enables linear models to capture non-linear relationships:
$$x_1, x_2 \rightarrow x_1, x_2, x_1^2, x_2^2, x_1 x_2$$
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)Categorical Feature Encoding
One-Hot Encoding
Creates a binary column for each category. Best for low-cardinality features (< ~20 unique values).
Ordinal Encoding
Maps categories to integers preserving order. Use only when a meaningful order exists (e.g., “low” < “medium” < “high”).
Target Encoding (Mean Encoding)
Replaces each category with the mean of the target variable within that category. Powerful for high-cardinality features but susceptible to leakage — use cross-validation or smoothing:
$$\text{encoded}(c) = \frac{n_c \cdot \bar{y}c + \lambda \cdot \bar{y}{global}}{n_c + \lambda}$$
where $\lambda$ is a smoothing factor and $n_c$ is the count of category $c$.
Frequency / Count Encoding
Replace categories with their frequency in the training set. Compact and useful when frequency itself is informative.
Embedding-Based Encoding
For very high-cardinality categories (user IDs, product IDs), train a learned embedding as part of the model (entity embeddings). Effective in neural networks and gradient boosting with categorical support (LightGBM, CatBoost).
Temporal Feature Engineering
Datetime features rarely improve models in raw form. Common transformations:
- Calendar features: year, month, day-of-month, day-of-week, hour, minute.
- Cyclical encoding: encode circular features (hour of day, day of week) as sine/cosine pairs to preserve periodicity: $$\sin\left(\frac{2\pi \cdot \text{hour}}{24}\right), \quad \cos\left(\frac{2\pi \cdot \text{hour}}{24}\right)$$
- Time since event: seconds/days since last purchase, last login, last failure.
- Rolling aggregates: rolling mean, max, std over a time window (e.g., purchases in last 7 days).
- Is weekend / Is holiday: binary indicators.
Text Feature Engineering
Beyond full NLP pipelines, basic text features are often useful:
- Bag of Words (BoW) / TF-IDF — Convert text to term frequency vectors.
- Character and word n-grams — Capture local patterns and morphology.
- Statistical features: character count, word count, sentence count, punctuation count, average word length.
- Embeddings: dense vectors from Word2Vec, FastText, or pre-trained LLMs (sentence-transformers).
Interaction Features
Manually or automatically create features from combinations of existing ones:
# Ratio features
df['price_per_sqft'] = df['price'] / df['area']
# Difference features
df['days_since_last_order'] = df['current_date'] - df['last_order_date']
# Flag features
df['is_large_transaction'] = (df['amount'] > 10000).astype(int)Tree-based models (Random Forest, XGBoost) automatically discover interactions during training — but explicitly providing known interactions can improve linear models significantly.
Feature Selection
After engineering, many features may be redundant, noisy, or irrelevant. Feature selection reduces overfitting and improves interpretability:
| Method | Type | Notes |
|---|---|---|
| Variance Threshold | Filter | Remove near-constant features |
| Correlation filter | Filter | Remove highly correlated pairs |
| Mutual Information | Filter | Rank by dependency with target |
| Recursive Feature Elimination (RFE) | Wrapper | Iteratively remove least important |
| LASSO (L1) regularization | Embedded | Shrinks irrelevant feature weights to 0 |
| Tree feature importance | Embedded | Impurity-based or permutation importance |
| SHAP values | Model-agnostic | Consistent, game-theoretic importance scores |
Automated Feature Engineering
Libraries like Featuretools automate the creation of aggregation and transformation features from relational data (deep feature synthesis). For tabular data, AutoML frameworks (AutoGluon, H2O AutoML) include automated feature engineering pipelines.
However, automated methods rarely replace domain-driven features that encode genuine causal or mechanistic understanding.
Feature Engineering for Deep Learning
Deep learning reduces the need for manual feature engineering for unstructured data (images, text, audio) — the network learns representations directly. However, for tabular data, feature engineering remains critical because:
- Deep learning models for tabular data are often not decisively better than gradient boosting.
- Domain-engineered features provide signal that is difficult to learn from raw data alone.
- Feature normalization and encoding are still required.
Best Practices
- Understand the data first — Explore distributions, check for leakage, identify domain patterns.
- Engineer from business intuition — Features derived from domain logic are often the most powerful.
- Never leak future information — Time-series features must use only past data relative to the prediction date.
- Validate on held-out data — Feature importance on training data is misleading; validate improvement on validation or test sets.
- Log transformations — Document every feature transformation for reproducibility and debugging.
- Iterative process — Feature engineering is an iterative loop: engineer → train → evaluate → refine.
Feature engineering is as much craft as science — the best practitioners combine statistical rigour with deep domain understanding to create representations that give models the best possible chance of learning the true underlying patterns.