Decision Trees and Ensemble Methods - A Practical Guide

Decision trees are one of the most intuitive and powerful machine learning algorithms. They form the foundation for ensemble methods that often achieve state-of-the-art performance. This guide explains how they work and when to use them.

Decision Trees: Learning by Asking Questions

How Decision Trees Work

A decision tree learns by recursively asking questions about the data features and splitting the data based on the answers. This creates a tree-like structure where:

• Root Node: The starting point with all data
• Internal Nodes: Decision points (if/then questions)
• Branches: Outcomes of decisions
• Leaf Nodes: Final predictions

Example: Should you play tennis?

                    Check Weather
                         |
            _____________|_____________
           /                           \
        Rainy                      Not Rainy
        /                               \
    Don't Play                    Check Wind
                                      |
                          ____________|____________
                         /                        \
                       Windy                    Not Windy
                       /                              \
                   Don't Play                    Check Humidity
                                                      |
                                         _____________|_____________
                                        /                           \
                                      High                          Low
                                      /                              \
                                  Don't Play                      Play

How Trees Make Decisions

Trees select features and split points to minimize impurity (uncertainty) in the resulting groups.

Common Split Criteria:

Gini Impurity (for classification):

• Measures probability of incorrectly classifying random element
• Ranges from 0 (pure) to 0.5 (maximum impurity)
• Formula: Gini = 1 - Σ(p_i)² where p_i is proportion of class i

Information Gain (Entropy):

• Measures reduction in uncertainty
• Based on information theory
• Gain = Initial Entropy - Weighted Average of Final Entropies

Variance Reduction (for regression):

• Minimizes squared differences from mean
• Similar to classification but with continuous values

Example: Building a Decision Tree for Iris Flower Classification

Starting with 150 iris flowers with features like sepal length, sepal width, petal length, petal width.

1. First Split: “Is petal length ≤ 2.4 cm?”

   • Yes → Very likely Setosa
   • No → Continue splitting

2. Second Split (for remaining): “Is petal width ≤ 1.7 cm?”

   • Yes → Likely Versicolor
   • No → Likely Virginica

The tree learns to ask the most informative questions first.

Advantages and Disadvantages of Decision Trees

Advantages

• Interpretable: Easy to understand and explain decisions
• No Preprocessing: Works with raw features, no normalization needed
• Handles Mixed Data: Works with numerical and categorical features
• Feature Importance: Shows which features matter most
• Fast: Quick predictions (O(log n) complexity)
• Non-parametric: No assumptions about data distribution

Disadvantages

• Overfitting: Prone to learning data noise with large trees
• Instability: Small data changes can produce very different trees
• High Variance: Different subsets of data produce different trees
• Biased with Imbalanced: Struggles with skewed class distributions
• Single Tree Limitations: Often not accurate enough alone

Ensemble Methods: Combining Trees for Better Performance

The limitations of single trees led to ensemble methods - combining multiple models for better predictions.

Random Forests

How It Works:

1. Create multiple decision trees
2. Each tree trained on random subset of data (bootstrap sample)
3. At each split, consider random subset of features
4. Average predictions (classification: majority vote, regression: average)

Why It’s Effective:

• Reduces overfitting through averaging
• Each tree sees different data variations
• Feature randomness adds diversity
• Parallel training possible

Hyperparameters:

• n_trees: Number of trees (typically 100-1000)
• max_depth: Maximum tree depth
• min_samples_split: Minimum samples to split node
• max_features: Number of features to consider per split

When to Use:

• Need robust, accurate predictions
• Want feature importance rankings
• Have moderate-sized datasets
• Can afford computational cost

Example Implementation Concept:

Random Forest for predicting house prices:
- Create 100 decision trees
- Each tree sees 80% of houses (random sample)
- Each split considers only 3-5 random features
- For new house: average predictions from all 100 trees
- More stable and accurate than single tree

Gradient Boosting

How It Works:

1. Start with simple model (often single tree)
2. Calculate residuals (errors) from first model
3. Train second model to predict residuals
4. Combine: prediction = tree1 + tree2
5. Repeat, each tree corrects previous errors

Why It’s Effective:

• Sequential learning focuses on hard cases
• Each tree specializes in correcting mistakes
• Often highest accuracy among tree-based methods

Popular Implementations:

• XGBoost (extremely fast and effective)
• LightGBM (lightens memory usage)
• CatBoost (handles categorical features well)

Hyperparameters:

• n_estimators: Number of boosting stages
• learning_rate: Contribution of each tree (smaller = slower but often better)
• max_depth: Tree depth
• subsample: Fraction of data per tree

When to Use:

• Need maximum accuracy
• Have time for hyperparameter tuning
• Using tabular/structured data
• Can invest computational resources

Adaboost

How It Works:

1. Train weak learner on all data
2. Increase weight on misclassified examples
3. Train new learner on weighted data
4. Repeat, each new learner focuses on hard examples
5. Combine with weighted voting

Characteristics:

• Reduces bias (like bagging reduces variance)
• Sensitive to noise and outliers
• Generally weaker than gradient boosting
• Easier to tune than gradient boosting

Voting and Stacking

Voting:

• Combine predictions from different algorithm types
• Each model votes (classification) or contributes (regression)
• Reduces overfitting from single model type

Stacking:

• Train multiple models as base learners
• Train meta-learner to combine base predictions
• Can capture complex relationships between models

Comparison of Ensemble Methods

Practical Guidelines

Start with Random Forest When:

• Working with structured data
• Want good accuracy without extensive tuning
• Need interpretability
• Have moderate computational resources
• Dataset is moderately sized

Move to Gradient Boosting When:

• Random Forest isn’t accurate enough
• You have time for hyperparameter tuning
• Computational resources available
• XGBoost/LightGBM libraries are accessible

Combine Approaches When:

• You need maximum possible accuracy
• Can afford ensemble of ensembles
• Have time and resources for tuning

Feature Importance from Trees

Trees reveal which features matter most:

Importance Calculation:

• How much each feature reduces impurity across all splits
• Features used in high-impact splits are more important
• Can guide feature engineering and data collection

Using Feature Importance:

• Identify key business drivers
• Reduce dimensionality by dropping low-importance features
• Validate against domain knowledge
• Guide data collection priorities

Conclusion

Decision trees provide interpretable, powerful building blocks. Ensemble methods amplify their strengths while mitigating weaknesses. Random forests offer good accuracy with reasonable simplicity. Gradient boosting methods achieve highest accuracy but require more tuning. Understanding these methods lets you build effective, accurate machine learning systems on structured data.