Skip to content

Decision Trees

What This Is

A decision tree is a model that recursively splits the feature space with axis-aligned thresholds. Each internal node says "if feature j ≤ t, go left, else go right"; each leaf stores a prediction — a class distribution or a mean target value.

The practical lesson is that a single decision tree is rarely the model you ship, but it is one of the sharpest diagnostic instruments you have. Its splits show which feature thresholds actually separate the classes, its depth reveals how much structure the data supports, and its failures reveal where you need a better feature or a different model.

When You Use It

  • as a transparent baseline whose splits you can read
  • to discover threshold-style structure that a linear model would miss
  • as the base learner inside random forests or gradient boosting — see Ensemble Methods
  • for features that mix categorical, numerical, and missing values
  • when coefficients are not enough — you need "if X and Y then Z" rules

Do Not Use It When

  • the relationship is globally linear — a single tree wastes capacity re-learning it with axis-aligned steps
  • you need calibrated probabilities from a single model — leaf-class-proportions are noisy
  • the training set is small and variance is high — move to the ensemble instead

Split Criteria

A split is chosen to make the child nodes more "pure" in label. The two standard impurity measures:

  • Gini impurity: 1 - Σ p_k² — probability of misclassifying a random sample drawn from the node under the node's label distribution. Minimized at 0 (pure).
  • Entropy: -Σ p_k log p_k — information content of the label distribution. Also minimized at 0 (pure).

At each candidate split, compute impurity(parent) - [n_left/n · impurity(left) + n_right/n · impurity(right)]. The split with the largest drop wins. This drop is called the information gain (entropy) or the Gini gain (Gini).

For regression trees, the impurity becomes variance (or MSE); the chosen split is the one that maximally reduces squared error in the children.

Gini vs. Entropy

In practice, Gini and entropy usually choose the same splits. Gini is slightly faster (no logarithm) and defaults in scikit-learn. Entropy is preferred when you want to interpret gains in information-theoretic units.

The choice between them is rarely the bottleneck — tree depth and regularization dominate.

Tooling

  • DecisionTreeClassifier
  • DecisionTreeRegressor
  • criterion="gini", "entropy", "log_loss"
  • max_depth
  • min_samples_split
  • min_samples_leaf
  • ccp_alpha for cost-complexity pruning
  • class_weight
  • export_text, plot_tree
  • feature_importances_

Regularization Is Depth Control

Unregularized trees can overfit perfectly on a training set — a leaf per point is achievable. Every practical tree is regularized:

  • max_depth — hard cap on tree depth (the most powerful knob)
  • min_samples_leaf — a leaf must contain at least this many training points
  • min_samples_split — a node must have at least this many points to even consider splitting
  • ccp_alpha — cost-complexity pruning; fits a full tree and prunes back weakest links

For a standalone tree, start with max_depth between 3 and 8 and tune from there. A tree deeper than ~10 on typical tabular data is usually memorizing.

Minimal Example

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=5, min_samples_leaf=20, random_state=0)
tree.fit(X_train, y_train)

Set random_state. Trees are deterministic given the same tie-breaking, but ties do happen; a fixed seed keeps runs comparable.

Reading A Tree

from sklearn.tree import export_text
print(export_text(tree, feature_names=list(X_train.columns)))

That text output is the single most useful debugging artifact a tree gives you. Read the first three splits. If they match the business logic, you have a real model. If they do not, you have a feature-engineering problem.

Feature Importance — With A Warning

tree.feature_importances_ reports how much each feature reduced impurity across the whole tree. Two caveats:

  • importance is biased toward high-cardinality features — a continuous feature with many split points will look more important than a categorical one even when its signal is equal
  • importance is correlation-sensitive — among two correlated features the tree will pick one and the other will look near-zero

For honest importance, prefer permutation importance (sklearn.inspection.permutation_importance), which measures accuracy drop when a feature's values are shuffled.

What To Inspect

  • the top three splits — do they align with domain knowledge
  • the training vs. validation gap across max_depth — a rising gap signals memorization
  • leaf sizes — are any leaves carrying only 1-2 examples
  • feature importances alongside permutation importance — do they agree
  • which class proportions live at the deepest leaves
  • whether a regression tree's residuals still show patterns (they almost always do, which is why boosting exists)

Failure Pattern

Reading a tree's feature_importances_ without cross-checking against permutation importance, and writing a report that says "feature X does not matter." X may be the most informative feature; it may just be collinear with Y and the tree happened to split on Y first.

A second failure pattern is shipping a single deep tree. A max_depth=None tree on tabular data almost always overfits; a shallow tree inside a random forest is the production answer.

A third failure pattern is trusting predicted probabilities from a tree leaf. Leaves with 4 training points and 3 positives report p=0.75, which is not a calibrated probability — it is a noisy leaf proportion.

Quick Checks

  1. Is max_depth set to something reasonable (or is ccp_alpha pruning)?
  2. Does the training accuracy greatly exceed validation accuracy?
  3. Do the top splits make sense under domain review?
  4. Do feature_importances_ and permutation importance agree?
  5. Are there leaves with fewer than ~20 samples, and is that a problem for the task?

Practice

  1. Train a DecisionTreeClassifier(max_depth=3) on a 2D toy problem and plot the axis-aligned regions.
  2. Sweep max_depth from 1 to 20 and plot train vs. validation accuracy. Identify the overfitting point.
  3. Compare criterion="gini" and criterion="entropy" on the same split. Explain the result.
  4. Print the tree with export_text and write the top three splits as human-readable rules.
  5. Add a copy of a feature (perfectly correlated) and compare feature_importances_ to permutation importance.
  6. Train a regression tree and inspect its residuals on the validation set.
  7. Explain the difference between min_samples_split and min_samples_leaf.
  8. Describe one case where Gini and entropy disagree and why it rarely matters.
  9. Explain what cost-complexity pruning (ccp_alpha) is actually minimizing.
  10. State one production reason to prefer a random forest over a single tree.

Longer Connection

Decision trees sit next to:

A single tree is an instrument, not a product. Its job is to show you the thresholds — the ensemble's job is to combine many of them into something stable.