# On Model Stability as a Function of Random Seed

arXiv: https://arxiv.org/abs/1909.10447 · HTML: https://mlsanity.com/paper/1909.10447/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 30 of 31)

- [Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits](https://mlsanity.com/paper/2411.18704/) (2024)
- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Nondeterminism and Instability in Neural Network Optimization](https://mlsanity.com/paper/2103.04514/) (2021)
- [On the Reproducibility of Neural Network Predictions](https://mlsanity.com/paper/2102.03349/) (2021)
- [A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation](https://mlsanity.com/paper/2207.07483/) (2022)
- [Revisiting the Fragility of Influence Functions](https://mlsanity.com/paper/2303.12922/) (2023)
- [Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery](https://mlsanity.com/paper/2105.10488/) (2021)
- [Latent State Models of Training Dynamics](https://mlsanity.com/paper/2308.09543/) (2023)
- [Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective](https://mlsanity.com/paper/2110.06256/) (2021)
- [FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling](https://mlsanity.com/paper/2303.00586/) (2023)
- [Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models](https://mlsanity.com/paper/2503.07329/) (2025)
- [Why only Micro-F1? Class Weighting of Measures for Relation Classification](https://mlsanity.com/paper/2205.09460/) (2022)
- [Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues](https://mlsanity.com/paper/2302.12691/) (2023)
- [Quantifying Inherent Randomness in Machine Learning Algorithms](https://mlsanity.com/paper/2206.12353/) (2022)
- [How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task](https://mlsanity.com/paper/2111.09612/) (2021)
- [Instability in clinical risk stratification models using deep learning](https://mlsanity.com/paper/2211.10828/) (2022)
- [ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP](https://mlsanity.com/paper/2308.02537/) (2023)
- [Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?](https://mlsanity.com/paper/2506.07871/) (2025)
- [An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering](https://mlsanity.com/paper/2201.01815/) (2022)
- [Link Prediction or Perdition: the Seeds of Instability in Knowledge Graph Embeddings](https://mlsanity.com/paper/2606.03365/) (2026)
- [Pseudo-Label Training and Model Inertia in Neural Machine Translation](https://mlsanity.com/paper/2305.11808/) (2023)
- [A Framework for Provably Stable and Consistent Training of Deep Feedforward Networks](https://mlsanity.com/paper/2305.12125/) (2023)
- [Beginning with You: Perceptual-Initialization Improves Vision-Language Representation and Alignment](https://mlsanity.com/paper/2505.14204/) (2025)
- [A solution to generalized learning from small training sets found in infants repeated visual experiences of individual objects](https://mlsanity.com/paper/2510.15060/) (2025)
- [ArchISMiner: A Framework for Automatic Mining of Architectural Issue-Solution Pairs from Online Developer Communities](https://mlsanity.com/paper/2510.21966/) (2025)
- [Sequence of Expert: Boosting Imitation Planners for Autonomous Driving through Temporal Alternation](https://mlsanity.com/paper/2512.13094/) (2025)
- [When Does Pairing Seeds Reduce Variance? Evidence from a Multi-Agent Economic Simulation](https://mlsanity.com/paper/2512.24145/) (2025)
- [Diagnostics for Individual-Level Prediction Instability in Machine Learning for Healthcare](https://mlsanity.com/paper/2603.00192/) (2026)
- [Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications](https://mlsanity.com/paper/2604.27987/) (2026)

## Abstract (arXiv)

In this paper, we focus on quantifying model stability as a function of random seed by investigating the effects of the induced randomness on model performance and the robustness of the model in general. We specifically perform a controlled study on the effect of random seeds on the behaviour of attention, gradient-based and surrogate model based (LIME) interpretations. Our analysis suggests that random seeds can adversely affect the consistency of models resulting in counterfactual interpretations. We propose a technique called Aggressive Stochastic Weight Averaging (ASWA)and an extension called Norm-filtered Aggressive Stochastic Weight Averaging (NASWA) which improves the stability of models over random seeds. With our ASWA and NASWA based optimization, we are able to improve the robustness of the original model, on average reducing the standard deviation of the model's performance by 72%.