# A Path to Simpler Models Starts With Noise

arXiv: https://arxiv.org/abs/2310.19726 · HTML: https://mlsanity.com/paper/2310.19726/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 56 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 30 of 32)

- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Measuring the Effects of Data Parallelism on Neural Network Training](https://mlsanity.com/paper/1811.03600/) (2018)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Label Noise SGD Provably Prefers Flat Global Minimizers](https://mlsanity.com/paper/2106.06530/) (2021)
- [Generalized and Scalable Optimal Sparse Decision Trees](https://mlsanity.com/paper/2006.08690/) (2020)
- [On the Existence of Simpler Machine Learning Models](https://mlsanity.com/paper/1908.01755/) (2019)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [On Counterfactual Explanations under Predictive Multiplicity](https://mlsanity.com/paper/2006.13132/) (2020)
- [Characterizing Fairness Over the Set of Good Models Under Selective Labels](https://mlsanity.com/paper/2101.00352/) (2021)
- [Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization](https://mlsanity.com/paper/1710.05179/) (2017)
- [Learning Certifiably Optimal Rule Lists for Categorical Data](https://mlsanity.com/paper/1704.01701/) (2017)
- [Rashomon Capacity: A Metric for Predictive Multiplicity in Classification](https://mlsanity.com/paper/2206.01295/) (2022)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [Exploring and Interacting with the Set of Good Sparse Generalized Additive Models](https://mlsanity.com/paper/2303.16047/) (2023)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [Ensembles of Locally Independent Prediction Models](https://mlsanity.com/paper/1911.01291/) (2019)
- [Characterizing the risk of fairwashing](https://mlsanity.com/paper/2106.07504/) (2021)
- [Nearly Optimal Algorithms for Level Set Estimation](https://mlsanity.com/paper/2111.01768/) (2021)
- [Robust Optimization using Machine Learning for Uncertainty Sets](https://mlsanity.com/paper/1407.1097/) (2014)
- [An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise](https://mlsanity.com/paper/1902.08234/) (2019)
- [TimberTrek: Exploring and Curating Sparse Decision Trees with Interactive Visualization](https://mlsanity.com/paper/2209.09227/) (2022)
- [Computing the Collection of Good Models for Rule Lists](https://mlsanity.com/paper/2204.11285/) (2022)

## Built on by (most cited first, 30 of 33)

- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Algorithmic Arbitrariness in Content Moderation](https://mlsanity.com/paper/2402.16979/) (2024)
- [Rashomon Sets for Prototypical-Part Networks: Editing Interpretable Models in Real-Time](https://mlsanity.com/paper/2503.01087/) (2025)
- [Optimal or Greedy Decision Trees? Revisiting their Objectives, Tuning, and Performance](https://mlsanity.com/paper/2409.12788/) (2024)
- [Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML](https://mlsanity.com/paper/2507.14744/) (2025)
- [Leveraging Predictive Equivalence in Decision Trees](https://mlsanity.com/paper/2506.14143/) (2025)
- [When and Where to Reset Matters for Long-Term Test-Time Adaptation](https://mlsanity.com/paper/2603.03796/) (2026)
- [Practical Attribution Guidance for Rashomon Sets](https://mlsanity.com/paper/2407.18482/) (2024)
- [On the Rashomon ratio of infinite hypothesis sets](https://mlsanity.com/paper/2404.17746/) (2024)
- [Unique Rashomon Sets for Robust Active Learning](https://mlsanity.com/paper/2503.06770/) (2025)
- [ElliCE: Efficient and Provably Robust Algorithmic Recourse via the Rashomon Sets](https://mlsanity.com/paper/2602.07674/) (2026)
- [Rashomon effect in Educational Research: Why More is Better Than One for Measuring the Importance of the Variables?](https://mlsanity.com/paper/2412.12115/) (2024)
- [Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity](https://mlsanity.com/paper/2505.16638/) (2025)
- [Leo Breiman, the Rashomon Effect, and the Occam Dilemma](https://mlsanity.com/paper/2507.03884/) (2025)
- [Towards the Formalization of a Trustworthy AI for Mining Interpretable Models explOiting Sophisticated Algorithms](https://mlsanity.com/paper/2510.20621/) (2025)
- [Parameter-Efficient Construction of the Rashomon Slice for Concept Bottleneck Models](https://mlsanity.com/paper/2511.19636/) (2025)
- [Why are there many equally good models? An Anatomy of the Rashomon Effect](https://mlsanity.com/paper/2601.06730/) (2026)
- [DIVERSE: Disagreement-Inducing Vector Evolution for Rashomon Set Exploration](https://mlsanity.com/paper/2601.20627/) (2026)
- [REALITrees: Rashomon Ensemble Active Learning for Interpretable Trees](https://mlsanity.com/paper/2603.22750/) (2026)
- [Measurement noise limits the advantage of nonlinear models over linear models in biomedical prediction](https://mlsanity.com/paper/2606.18420/) (2026)
- [Natural Learning](https://mlsanity.com/paper/2404.05903/) (2024)
- [VAR: Visual Analysis for Rashomon Set of Machine Learning Models' Performance](https://mlsanity.com/paper/2507.22556/) (2025)
- [Data as a Lever: A Neighbouring Datasets Perspective on Predictive Multiplicity](https://mlsanity.com/paper/2510.21303/) (2025)
- [The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learning](https://mlsanity.com/paper/2511.21799/) (2025)
- [Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series](https://mlsanity.com/paper/2603.02620/) (2026)
- [Model Multiplicity and Predictive Arbitrariness in Recidivism Risk Assessment](https://mlsanity.com/paper/2606.02198/) (2026)
- [Is Model Instability just Noise to be Tolerated or a Property that can be Managed?](https://mlsanity.com/paper/2607.10420/) (2026)
- [Do Judges Behave Like Algorithms?](https://mlsanity.com/paper/2608.10400/) (2026)
- [TetrisCNN for interpretable detection of phases of matter from experimental quantum simulator data](https://mlsanity.com/paper/2609.20693/) (2026)

## Abstract (arXiv)

The Rashomon set is the set of models that perform approximately equally well on a given dataset, and the Rashomon ratio is the fraction of all models in a given hypothesis space that are in the Rashomon set. Rashomon ratios are often large for tabular datasets in criminal justice, healthcare, lending, education, and in other areas, which has practical implications about whether simpler models can attain the same level of accuracy as more complex models. An open question is why Rashomon ratios often tend to be large. In this work, we propose and study a mechanism of the data generation process, coupled with choices usually made by the analyst during the learning process, that determines the size of the Rashomon ratio. Specifically, we demonstrate that noisier datasets lead to larger Rashomon ratios through the way that practitioners train models. Additionally, we introduce a measure called pattern diversity, which captures the average difference in predictions between distinct classification patterns in the Rashomon set, and motivate why it tends to increase with label noise. Our results explain a key aspect of why simpler models often tend to perform as well as black box models on complex, noisier datasets.