# Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles

arXiv: https://arxiv.org/abs/2109.10432 · HTML: https://mlsanity.com/paper/2109.10432/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 76 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Builds on (most cited first, 27 of 27)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Multiaccuracy: Black-Box Post-Processing for Fairness in Classification](https://mlsanity.com/paper/1805.12317/) (2018)
- [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020)
- [Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach](https://mlsanity.com/paper/1610.03425/) (2016)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [Learning explanations that are hard to vary](https://mlsanity.com/paper/2009.00329/) (2020)
- [Distributionally Robust Logistic Regression](https://mlsanity.com/paper/1509.09259/) (2015)
- [Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions](https://mlsanity.com/paper/1707.06422/) (2017)
- [Invariance, Causality and Robustness](https://mlsanity.com/paper/1812.08233/) (2018)
- [Linear unit-tests for invariance discovery](https://mlsanity.com/paper/2102.10867/) (2021)
- [Causal Regularization](https://mlsanity.com/paper/1906.12179/) (2019)
- [Regularizing towards Causal Invariance: Linear Models with Proxies](https://mlsanity.com/paper/2103.02477/) (2021)
- [Versatile Verification of Tree Ensembles](https://mlsanity.com/paper/2010.13880/) (2020)
- [Discovering Reliable Causal Rules](https://mlsanity.com/paper/2009.02728/) (2020)
- [Causal Transfer Random Forest: Combining Logged Data and Randomized Experiments for Robust Prediction](https://mlsanity.com/paper/2010.08710/) (2020)
- [Efficiently Discovering Locally Exceptional yet Globally Representative Subgroups](https://mlsanity.com/paper/1709.07941/) (2017)

## Built on by (most cited first, 4 of 4)

- [Provable Domain Generalization via Invariant-Feature Subspace Recovery](https://mlsanity.com/paper/2201.12919/) (2022)
- [Vision Checklist: Towards Testable Error Analysis of Image Models to Help System Designers Interrogate Model Capabilities](https://mlsanity.com/paper/2201.11674/) (2022)
- [Invariant-Feature Subspace Recovery: A New Class of Provable Domain Generalization Algorithms](https://mlsanity.com/paper/2311.00966/) (2023)
- [Conformalized Exceptional Model Mining: Telling Where Your Model Performs (Not) Well](https://mlsanity.com/paper/2508.15569/) (2025)

## Abstract (arXiv)

Local decision rules are commonly understood to be more explainable, due to the local nature of the patterns involved. With numerical optimization methods such as gradient boosting, ensembles of local decision rules can gain good predictive performance on data involving global structure. Meanwhile, machine learning models are being increasingly used to solve problems in high-stake domains including healthcare and finance. Here, there is an emerging consensus regarding the need for practitioners to understand whether and how those models could perform robustly in the deployment environments, in the presence of distributional shifts. Past research on local decision rules has focused mainly on maximizing discriminant patterns, without due consideration of robustness against distributional shifts. In order to fill this gap, we propose a new method to learn and ensemble local decision rules, that are robust both in the training and deployment environments. Specifically, we propose to leverage causal knowledge by regarding the distributional shifts in subpopulations and deployment environments as the results of interventions on the underlying system. We propose two regularization terms based on causal knowledge to search for optimal and stable rules. Experiments on both synthetic and benchmark datasets show that our method is effective and robust against distributional shifts in multiple environments.