mlsanityData to 13 Oct 2026

Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning

arXiv 2501.14959 ↗First posted 24 Jan 2025cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Algorithmic modeling relies on limited information in data to extrapolate outcomes for unseen scenarios, often embedding an element of arbitrariness in its decisions. A perspective on this arbitrariness that has recently gained interest is multiplicity-the study of arbitrariness across a set of "good models", i.e., those likely to be deployed in practice. In this work, we systemize the literature on multiplicity by: (a) formalizing the terminology around model design choices and their contribution to arbitrariness, (b) expanding the definition of multiplicity to incorporate underrepresented forms beyond just predictions and explanations, (c) clarifying the distinction between multiplicity and other lenses of arbitrariness, i.e., uncertainty and variance, and (d) distilling the benefits and potential risks of multiplicity into overarching trends, situating it within the broader landscape of responsible AI. We conclude by identifying open research questions and highlighting emerging trends in this young but rapidly growing area of research.

Key facts. Cited by 17 papers in the map (data through 13 Oct 2026). Builds on 79 of its 173 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

79 of 173 references.

References · 79

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,336
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,340
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,944
  4. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 910
  5. 2020On Adaptive Attacks to Adversarial Example Defensescited by 554
  6. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 489
  7. 2019On Evaluating Adversarial Robustnesscited by 447
  8. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 368
  9. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 365
  10. 2019Measurement and Fairnesscited by 303
  11. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  12. 2021Data Augmentation Can Improve Robustnesscited by 193
38 more
  1. 2019Evaluating model calibration in classificationcited by 165
  2. 2019Predictive Multiplicity in Classificationcited by 163
  3. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  4. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 151
  5. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 127
  6. 2022Bias Mitigation for Machine Learning Classifiers: A Comprehensive Surveycited by 125
  7. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  8. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  9. 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 89
  10. 2022Exploring the Whole Rashomon Set of Sparse Decision Treescited by 81
  11. 2020Do Wider Neural Networks Really Help Adversarial Robustness?cited by 79
  12. 2021Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer visioncited by 79
  13. 2020General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Modelscited by 74
  14. 2021Characterizing Fairness Over the Set of Good Models Under Selective Labelscited by 65
  15. 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
  16. 2021Directional Bias Amplificationcited by 54
  17. 2024Amazing Things Come From Having Many Good Modelscited by 54
  18. 2022Rashomon Capacity: A Metric for Predictive Multiplicity in Classificationcited by 47
  19. 2023Robust Counterfactual Explanations for Neural Networks With Probabilistic Guaranteescited by 45
  20. 2020Algorithmic Recourse in the Wild: Understanding the Impact of Data and Model Shiftscited by 34
  21. 2023A Path to Simpler Models Starts With Noisecited by 33
  22. 2023Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classificationcited by 31
  23. 2023The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importancecited by 29
  24. 2017Active Learning Using Uncertainty Informationcited by 28
  25. 2023On The Impact of Machine Learning Randomness on Group Fairnesscited by 26
  26. 2023An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learningcited by 25
  27. 2020The Grammar of Interactive Explanatory Model Analysiscited by 24
  28. 2021Selective Ensembles for Consistent Predictionscited by 23
  29. 2022FasterRisk: Fast and Accurate Interpretable Risk Scorescited by 21
  30. 2021Hyperparameter Optimization Is Deceiving Us, and How to Stop Itcited by 20
  31. 2024Position: Explain to Question not to Justifycited by 20
  32. 2019Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Modelscited by 19
  33. 2023Arbitrary Decisions are a Hidden Cost of Differentially Private Trainingcited by 19
  34. 2021Loss-Aversively Fair Classificationcited by 18
  35. 2024Dropout-Based Rashomon Set Exploration for Efficient Predictive Multiplicity Estimationcited by 18
  36. 2023The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictionscited by 17
  37. 2024Predictive Churn with the Set of Good Modelscited by 15
  38. 2014Robust Optimization using Machine Learning for Uncertainty Setscited by 14

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 6; 2026: 11.

17 papers.

Sort:

By area:

  1. 2025Observational Multiplicitycited by 2
  2. 2026ElliCE: Efficient and Provably Robust Algorithmic Recourse via the Rashomon Setscited by 2
  3. 2026Using predictive multiplicity to measure individual performance within the AI Actcited by 2
  4. 2025Semivalue-based data valuation is arbitrary and gameablecited by 1
  5. 2025Argumentative Ensembling for Robust Recourse under Model Multiplicitycited by 1
  6. 2025Parameter-Efficient Construction of the Rashomon Slice for Concept Bottleneck Modelscited by 1
  7. 2026Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicitycited by 1
  8. 2026Robust Ambiguity Detection (RAD) From Model- and Feature-Space Consistencycited by 1
  9. 2025Data as a Lever: A Neighbouring Datasets Perspective on Predictive Multiplicitycited by 0
  10. 2025The Double-Edged Nature of the Rashomon Set for Trustworthy Machine Learningcited by 0
  11. 2026Generating Diverse TSP Tours via a Combination of Graph Pointer Network and Dispersioncited by 0
  12. 2026Rashomon Sets and Model Multiplicity in Federated Learningcited by 0
5 more
  1. 2026An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairnesscited by 0
  2. 2026Explainable AI Isn't Enough! Rethinking Algorithmic Contestabilitycited by 0
  3. 2026Model Multiplicity and Predictive Arbitrariness in Recidivism Risk Assessmentcited by 0
  4. 2026The Impact of Stochasticity on the Rashomon Effect in Machine Learningcited by 0
  5. 2026Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packagescited by 0