mlsanityData to 7 Oct 2026

Reproducibility in Optimization: Theoretical Framework and Limits

arXiv 2202.04598 ↗First posted 9 Feb 2022math.OC cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

We initiate a formal study of reproducibility in optimization. We define a quantitative measure of reproducibility of optimization procedures in the face of noisy or error-prone operations such as inexact or stochastic gradient computations or inexact initialization. We then analyze several convex optimization settings of interest such as smooth, non-smooth, and strongly-convex objective functions and establish tight bounds on the limits of reproducibility in each setting. Our analysis reveals a fundamental trade-off between computation and reproducibility: more computation is necessary (and sufficient) for better reproducibility.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 25 of its 49 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

25 of 49 references.

References · 25

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2014Convex Optimization: Algorithms and Complexitycited by 569
  4. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  5. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  6. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  7. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  8. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  9. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  10. 2018Large scale distributed neural network training through online distillationcited by 144
  11. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  12. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
13 more
  1. 2021Nondeterminism and Instability in Neural Network Optimizationcited by 47
  2. 2022Reproducibility in Learningcited by 44
  3. 2019Analyzing the Role of Model Uncertainty for Electronic Health Recordscited by 41
  4. 2021On the Sample Complexity of Stability Constrained Imitation Learningcited by 38
  5. 2019The Complexity of Making the Gradient Small in Stochastic Convex Optimizationcited by 33
  6. 2021On the Reproducibility of Neural Network Predictionscited by 28
  7. 2018On Acceleration with Noise-Corrupted Gradientscited by 20
  8. 2021Algorithmic Instabilities of Accelerated Gradient Descentcited by 16
  9. 2020Smooth activations and reproducibility in deep networkscited by 15
  10. 2020Anti-Distillation: Improving reproducibility of deep networkscited by 13
  11. 2018Robust Accelerated Gradient Methods for Smooth Strongly Convex Functionscited by 12
  12. 2021Synthesizing Irreproducibility in Deep Networkscited by 9
  13. 2020Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systemscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 8; 2024: 3; 2025: 3; 2026: 3.

21 papers.

Sort:

  1. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  2. 2022Replicable Banditscited by 22
  3. 2023Replicable Reinforcement Learningcited by 21
  4. 2023Replicable Clusteringcited by 19
  5. 2023Statistical Indistinguishability of Learning Algorithmscited by 17
  6. 2023List and Certificate Complexities in Replicable Learningcited by 13
  7. 2023Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Testscited by 11
  8. 2023Replicability and stability in learningcited by 8
  9. 2024What Do Machine Learning Researchers Mean by "Reproducible"?cited by 8
  10. 2022A Siren Song of Open Source Reproducibilitycited by 7
  11. 2024Replicable Online Learningcited by 7
  12. 2025On Corruption-Robustness in Performative Reinforcement Learningcited by 6
9 more
  1. 2025Replicable Reinforcement Learning with Linear Function Approximationcited by 6
  2. 2023Towards Inferential Reproducibility of Machine Learning Researchcited by 4
  3. 2023Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimizationcited by 3
  4. 2022Real World Large Scale Recommendation Systems Reproducibility and Smooth Activationscited by 2
  5. 2026Replicable Constrained Banditscited by 2
  6. 2024Generalizability of experimental studiescited by 0
  7. 2025Fractal basin geometry and the limits to predictability and reproducibility in deep learningcited by 0
  8. 2026Calibrating Adaptive Smoothing Methods for Freeway Traffic Reconstructioncited by 0
  9. 2026Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivitycited by 0