# Hierarchical Shrinkage: improving the accuracy and interpretability of tree-based methods

arXiv: https://arxiv.org/abs/2202.00858 · HTML: https://mlsanity.com/paper/2202.00858/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 60 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 17 of 17)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model](https://mlsanity.com/paper/1511.01644/) (2015)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [Generalized and Scalable Optimal Sparse Decision Trees](https://mlsanity.com/paper/2006.08690/) (2020)
- [PMLB v1.0: An open source dataset collection for benchmarking machine learning methods](https://mlsanity.com/paper/2012.00058/) (2020)
- [Iterative Random Forests to detect predictive and stable high-order interactions](https://mlsanity.com/paper/1706.08457/) (2017)
- [Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success](https://mlsanity.com/paper/1911.00190/) (2019)
- [Data-driven Advice for Applying Machine Learning to Bioinformatics Problems](https://mlsanity.com/paper/1708.05070/) (2017)
- [Interpolation can hurt robust generalization even when there is no noise](https://mlsanity.com/paper/2108.02883/) (2021)
- [A cautionary tale on fitting decision trees to data from additive models: generalization lower bounds](https://mlsanity.com/paper/2110.09626/) (2021)
- [Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning](https://mlsanity.com/paper/2102.12328/) (2021)

## Built on by (most cited first, 19 of 19)

- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Augmenting Interpretable Models with LLMs during Training](https://mlsanity.com/paper/2209.11799/) (2022)
- [Augmented balancing weights as linear regression](https://mlsanity.com/paper/2304.14545/) (2023)
- [ED-Copilot: Reduce Emergency Department Wait Time with Language Model Diagnostic Assistance](https://mlsanity.com/paper/2402.13448/) (2024)
- [Learning a Decision Tree Algorithm with Transformers](https://mlsanity.com/paper/2402.03774/) (2024)
- [On the Convergence of CART under Sufficient Impurity Decrease Condition](https://mlsanity.com/paper/2310.17114/) (2023)
- [Improving Prototypical Visual Explanations with Reward Reweighing, Reselection, and Retraining](https://mlsanity.com/paper/2307.03887/) (2023)
- [BELLATREX: Building Explanations through a LocaLly AccuraTe Rule EXtractor](https://mlsanity.com/paper/2203.15511/) (2022)
- [Error Reduction from Stacked Regressions](https://mlsanity.com/paper/2309.09880/) (2023)
- [STAND: Self-Aware Precondition Induction for Interactive Task Learning](https://mlsanity.com/paper/2409.07653/) (2024)
- [Surrogate Interpretable Graph for Random Decision Forests](https://mlsanity.com/paper/2506.01988/) (2025)
- [Agentic-imodels: Evolving agentic interpretability tools via autoresearch](https://mlsanity.com/paper/2605.03808/) (2026)
- [Group Probability-Weighted Tree Sums for Interpretable Modeling of Heterogeneous Data](https://mlsanity.com/paper/2205.15135/) (2022)
- [Explaining Patterns in Data with Language Models via Interpretable Autoprompting](https://mlsanity.com/paper/2210.01848/) (2022)
- [Bayesian post-hoc regularization of random forests](https://mlsanity.com/paper/2306.03702/) (2023)
- [Empowering Decision Trees via Shape Function Branching](https://mlsanity.com/paper/2510.19040/) (2025)
- [On Imbalanced Regression with Hoeffding Trees](https://mlsanity.com/paper/2602.22101/) (2026)
- [Kernel of Partition Paths: A Unified Representation for Tree Ensembles](https://mlsanity.com/paper/2606.18853/) (2026)
- [Coding LLMs Can Be Harnessed to Learn Interpretable Models from Data](https://mlsanity.com/paper/2610.14006/) (2026)

## Abstract (arXiv)

Tree-based models such as decision trees and random forests (RF) are a cornerstone of modern machine-learning practice. To mitigate overfitting, trees are typically regularized by a variety of techniques that modify their structure (e.g. pruning). We introduce Hierarchical Shrinkage (HS), a post-hoc algorithm that does not modify the tree structure, and instead regularizes the tree by shrinking the prediction over each node towards the sample means of its ancestors. The amount of shrinkage is controlled by a single regularization parameter and the number of data points in each ancestor. Since HS is a post-hoc method, it is extremely fast, compatible with any tree growing algorithm, and can be used synergistically with other regularization techniques. Extensive experiments over a wide variety of real-world datasets show that HS substantially increases the predictive performance of decision trees, even when used in conjunction with other regularization techniques. Moreover, we find that applying HS to each tree in an RF often improves accuracy, as well as its interpretability by simplifying and stabilizing its decision boundaries and SHAP values. We further explain the success of HS in improving prediction performance by showing its equivalence to ridge regression on a (supervised) basis constructed of decision stumps associated with the internal nodes of a tree. All code and models are released in a full-fledged package available on Github (github.com/csinva/imodels)