# Orthogonal Gradient Boosting for Simpler Additive Rule Ensembles

arXiv: https://arxiv.org/abs/2402.15691 · HTML: https://mlsanity.com/paper/2402.15691/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 38 references that are in the map.
- First posted on arXiv 24 Feb 2024.

## Builds on (most cited first, 8 of 8)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Boolean Decision Rules via Column Generation](https://mlsanity.com/paper/1805.09901/) (2018)
- [Generalized Linear Rule Models](https://mlsanity.com/paper/1906.01761/) (2019)
- [Interpretable Random Forests via Rule Extraction](https://mlsanity.com/paper/2004.14841/) (2020)
- [Identifying Consistent Statements about Numerical Data with Dispersion-Corrected Subgroup Discovery](https://mlsanity.com/paper/1701.07696/) (2017)
- [Better Short than Greedy: Interpretable Models through Optimal Rule Boosting](https://mlsanity.com/paper/2101.08380/) (2021)

## Built on by (most cited first, 2 of 2)

- [Interpretable Representation Learning for Additive Rule Ensembles](https://mlsanity.com/paper/2506.20927/) (2025)
- [Sparse Oblique Rule Boosting for Simpler Additive Rule Ensembles](https://mlsanity.com/paper/2609.06426/) (2026)

## Abstract (arXiv)

Gradient boosting of prediction rules is an efficient approach to learn potentially interpretable yet accurate probabilistic models. However, actual interpretability requires to limit the number and size of the generated rules, and existing boosting variants are not designed for this purpose. Though corrective boosting refits all rule weights in each iteration to minimise prediction risk, the included rule conditions tend to be sub-optimal, because commonly used objective functions fail to anticipate this refitting. Here, we address this issue by a new objective function that measures the angle between the risk gradient vector and the projection of the condition output vector onto the orthogonal complement of the already selected conditions. This approach correctly approximate the ideal update of adding the risk gradient itself to the model and favours the inclusion of more general and thus shorter rules. As we demonstrate using a wide range of prediction tasks, this significantly improves the comprehensibility/accuracy trade-off of the fitted ensemble. Additionally, we show how objective values for related rule conditions can be computed incrementally to avoid any substantial computational overhead of the new method.