mlsanityData to 7 Oct 2026

Predictive Churn with the Set of Good Models

arXiv 2402.07745 ↗First posted 12 Feb 2024cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Issues can arise when research focused on fairness, transparency, or safety is conducted separately from research driven by practical deployment concerns and vice versa. This separation creates a growing need for translational work that bridges the gap between independently studied concepts that may be fundamentally related. This paper explores connections between two seemingly unrelated concepts of predictive inconsistency that share intriguing parallels. The first, known as predictive multiplicity, occurs when models that perform similarly (e.g., nearly equivalent training loss) produce conflicting predictions for individual samples. This concept is often emphasized in algorithmic fairness research as a means of promoting transparency in ML model development. The second concept, predictive churn, examines the differences in individual predictions before and after model updates, a key challenge in deploying ML models in consumer-facing applications. We present theoretical and empirical results that uncover links between these previously disconnected concepts.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 61 of its 98 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

61 of 98 references.

References · 61

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  7. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2017Deep Mutual Learningcited by 621
  10. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  11. 2015Weight Uncertainty in Neural Networkscited by 554
  12. 2015Towards Open Set Deep Networkscited by 533
38 more
  1. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  2. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  4. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  5. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 364
  6. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  7. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  8. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  9. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  10. 2019Predictive Multiplicity in Classificationcited by 162
  11. 2020Characterising Bias in Compressed Modelscited by 162
  12. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  13. 2018Large scale distributed neural network training through online distillationcited by 144
  14. 2017DOC: Deep Open Classification of Text Documentscited by 109
  15. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  16. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  17. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  18. 2014Manifold Gaussian Processes for Regressioncited by 79
  19. 2021Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Testscited by 77
  20. 2020Towards Backward-Compatible Representation Learningcited by 74
  21. 2020On Counterfactual Explanations under Predictive Multiplicitycited by 73
  22. 2018Collaborative Learning for Deep Neural Networkscited by 69
  23. 2016Satisfying Real-world Goals with Dataset Constraintscited by 68
  24. 2021Characterizing Fairness Over the Set of Good Models Under Selective Labelscited by 65
  25. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
  26. 2018A Swiss Army Infinitesimal Jackknifecited by 51
  27. 2022Rashomon Capacity: A Metric for Predictive Multiplicity in Classificationcited by 46
  28. 2022Predictive Multiplicity in Probabilistic Classificationcited by 44
  29. 2019Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learningcited by 37
  30. 2021Leave-one-out Unfairnesscited by 36
  31. 2023Exploring and Interacting with the Set of Good Sparse Generalized Additive Modelscited by 30
  32. 2023The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importancecited by 29
  33. 2023On The Impact of Machine Learning Randomness on Group Fairnesscited by 26
  34. 2021Hyperparameter Optimization Is Deceiving Us, and How to Stop Itcited by 20
  35. 2019Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Modelscited by 19
  36. 2023Arbitrary Decisions are a Hidden Cost of Differentially Private Trainingcited by 19
  37. 2021Loss-Aversively Fair Classificationcited by 18
  38. 2021Churn Reduction via Distillationcited by 17

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 4.

15 papers.

Sort:

  1. 2025Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learningcited by 17
  2. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  3. 2024What Constitutes a Less Discriminatory Algorithm?cited by 8
  4. 2025Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoMLcited by 6
  5. 2024The Cost of Arbitrariness for Individuals: Examining the Legal and Technical Challenges of Model Multiplicitycited by 4
  6. 2025Regretful Decisions under Label Noisecited by 1
  7. 2026Model Agreement via Anchoringcited by 1
  8. 2026The Subjectivity of Monoculturecited by 1
  9. 2026Robust Ambiguity Detection (RAD) From Model- and Feature-Space Consistencycited by 1
  10. 2024From XAI to MLOps: Explainable Concept Drift Detection with Profile Drift Detectioncited by 0
  11. 2025Reconciling Predictive Multiplicity in Practicecited by 0
  12. 2025What Must a Fairness Audit Report When Demographic Data Is Incomplete?cited by 0
3 more
  1. 2025VAR: Visual Analysis for Rashomon Set of Machine Learning Models' Performancecited by 0
  2. 2025Data as a Lever: A Neighbouring Datasets Perspective on Predictive Multiplicitycited by 0
  3. 2026Analyzing the Temporal Factors for Anxiety and Depression Symptoms with the Rashomon Perspectivecited by 0