# Covariance-Driven Regression Trees: Reducing Overfitting in CART

arXiv: https://arxiv.org/abs/2601.07281 · HTML: https://mlsanity.com/paper/2601.07281/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Large Scale Prediction with Decision Trees](https://mlsanity.com/paper/2104.13881/) (2021)
- [Narrowing the Gap: Random Forests In Theory and In Practice](https://mlsanity.com/paper/1310.1415/) (2013)
- [Stratification Trees for Adaptive Randomization in Randomized Controlled Trials](https://mlsanity.com/paper/1806.05127/) (2018)
- [Sparse learning with CART](https://mlsanity.com/paper/2006.04266/) (2020)
- [A cautionary tale on fitting decision trees to data from additive models: generalization lower bounds](https://mlsanity.com/paper/2110.09626/) (2021)
- [On the Pointwise Behavior of Recursive Partitioning and Its Implications for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.10805/) (2022)
- [Oracle Inequalities for High-dimensional Prediction](https://mlsanity.com/paper/1608.00624/) (2016)
- [Nonparametric Variable Screening with Optimal Decision Stumps](https://mlsanity.com/paper/2011.02683/) (2020)

## Abstract (arXiv)

Decision trees are powerful machine learning algorithms, widely used in fields such as economics and medicine for their simplicity and interpretability. However, decision trees such as CART are prone to overfitting, especially when grown deep or the sample size is small. Conventional methods to reduce overfitting include pre-pruning and post-pruning, which constrain the growth of uninformative branches. In this paper, we propose a complementary approach by introducing a covariance-driven splitting criterion for regression trees (CovRT). This method is more robust to overfitting than the empirical risk minimization criterion used in CART, as it produces more balanced and stable splits and more effectively identifies covariates with true signals. We establish an oracle inequality of CovRT and prove that its predictive accuracy is comparable to that of CART in high-dimensional settings. We find that CovRT achieves superior prediction accuracy compared to CART in both simulations and real-world tasks.