# Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting

arXiv: https://arxiv.org/abs/2602.05371 · HTML: https://mlsanity.com/paper/2602.05371/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Builds on (most cited first, 5 of 5)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Convex Geometry and Duality of Over-parameterized Neural Networks](https://mlsanity.com/paper/2002.11219/) (2020)
- [Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descent](https://mlsanity.com/paper/2102.07567/) (2021)
- [Convergence Rates of Oblique Regression Trees for Flexible Function Libraries](https://mlsanity.com/paper/2210.14429/) (2022)
- [Vanilla Gradient Descent for Oblique Decision Trees](https://mlsanity.com/paper/2408.09135/) (2024)

## Built on by (most cited first, 1 of 1)

- [Hinge Regression Trees and HRT-Boost: Newton-Optimized Oblique Learning for Compact Tabular Models](https://mlsanity.com/paper/2605.23422/) (2026)

## Abstract (arXiv)

Oblique decision trees combine the transparency of trees with the power of multivariate decision boundaries, but learning high-quality oblique splits is NP-hard, and practical methods still rely on slow search or theory-free heuristics. We present the Hinge Regression Tree (HRT), which reframes each split as a non-linear least-squares problem over two linear predictors whose max/min envelope induces ReLU-like expressive power. The resulting alternating fitting procedure is exactly equivalent to a damped Newton (Gauss-Newton) method within fixed partitions. We analyze this node-level optimization and, for a backtracking line-search variant, prove that the local objective decreases monotonically and converges; in practice, both fixed and adaptive damping yield fast, stable convergence and can be combined with optional ridge regularization. We further prove that HRT's model class is a universal approximator with an explicit O(δ2) approximation rate, and show on synthetic and real-world benchmarks that it matches or outperforms single-tree baselines with more compact structures.