# Ranking Perspective for Tree-based Methods with Applications to Symbolic Feature Selection

arXiv: https://arxiv.org/abs/2410.02623 · HTML: https://mlsanity.com/paper/2410.02623/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Discussion on "Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects" by Hahn, Murray and Carvalho](https://mlsanity.com/paper/2108.02836/) (2021)
- [Large Scale Prediction with Decision Trees](https://mlsanity.com/paper/2104.13881/) (2021)
- [On Theory for BART](https://mlsanity.com/paper/1810.00787/) (2018)
- [Variable selection for BART: An application to gene regulation](https://mlsanity.com/paper/1310.4887/) (2013)
- [A Mixing Time Lower Bound for a Simplified Version of BART](https://mlsanity.com/paper/2210.09352/) (2022)
- [Sharded Bayesian Additive Regression Trees](https://mlsanity.com/paper/2306.00361/) (2023)
- [Variable Selection Using Bayesian Additive Regression Trees](https://mlsanity.com/paper/2112.13998/) (2021)
- [Classification Trees for Imbalanced and Sparse Data: Surface-to-Volume Regularization](https://mlsanity.com/paper/2004.12293/) (2020)
- [Assessing variable activity for Bayesian regression trees](https://mlsanity.com/paper/2005.13622/) (2020)
- [Frontal Slice Approaches for Tensor Linear Systems](https://mlsanity.com/paper/2408.13547/) (2024)
- [Learning Asymmetric and Local Features in Multi-Dimensional Data through Wavelets with Recursive Partitioning](https://mlsanity.com/paper/1711.00789/) (2017)
- [On Tree-based Methods for Similarity Learning](https://mlsanity.com/paper/1906.09243/) (2019)
- [Efficient Decision Trees for Tensor Regressions](https://mlsanity.com/paper/2408.01926/) (2024)

## Built on by (most cited first, 1 of 1)

- [Ab Initio Nonparametric Variable Selection for Scalable Symbolic Regression with Large p](https://mlsanity.com/paper/2410.13681/) (2024)

## Abstract (arXiv)

Tree-based methods are powerful nonparametric techniques in statistics and machine learning. However, their effectiveness, particularly in finite-sample settings, is not fully understood. Recent applications have revealed their surprising ability to distinguish transformations (which we call symbolic feature selection) that remain obscure under current theoretical understanding. This work provides a finite-sample analysis of tree-based methods from a ranking perspective. We link oracle partitions in tree methods to response rankings at local splits, offering new insights into their finite-sample behavior in regression and feature selection tasks. Building on this local ranking perspective, we extend our analysis in two ways: (i) We examine the global ranking performance of individual trees and ensembles, including Classification and Regression Trees (CART) and Bayesian Additive Regression Trees (BART), providing finite-sample oracle bounds, ranking consistency, and posterior contraction results. (ii) Inspired by the ranking perspective, we propose concordant divergence statistics T0 to evaluate symbolic feature mappings and establish their properties. Numerical experiments demonstrate the competitive performance of these statistics in symbolic feature selection tasks compared to existing methods.