# Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models

arXiv: https://arxiv.org/abs/2503.07329 · HTML: https://mlsanity.com/paper/2503.07329/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision](https://mlsanity.com/paper/2109.08203/) (2021)
- [Coverage-based Example Selection for In-Context Learning](https://mlsanity.com/paper/2305.14907/) (2023)
- [The Impact of Initialization on LoRA Finetuning Dynamics](https://mlsanity.com/paper/2406.08447/) (2024)
- [On Model Stability as a Function of Random Seed](https://mlsanity.com/paper/1909.10447/) (2019)
- [On The Impact of Machine Learning Randomness on Group Fairness](https://mlsanity.com/paper/2307.04138/) (2023)
- [We need to talk about random seeds](https://mlsanity.com/paper/2210.13393/) (2022)
- [How many images do I need? Understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoring](https://mlsanity.com/paper/2010.08186/) (2020)

## Built on by (most cited first, 10 of 10)

- [The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?](https://mlsanity.com/paper/2601.23045/) (2026)
- [GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Libraries](https://mlsanity.com/paper/2508.00033/) (2025)
- [Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models](https://mlsanity.com/paper/2604.13413/) (2026)
- [On the Extreme Variance of Certified Local Robustness Across Model Seeds](https://mlsanity.com/paper/2601.13303/) (2026)
- [Diagnostics for Individual-Level Prediction Instability in Machine Learning for Healthcare](https://mlsanity.com/paper/2603.00192/) (2026)
- [Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications](https://mlsanity.com/paper/2604.27987/) (2026)
- [GRAIN: Group Aggregation via Min-Norm Objective](https://mlsanity.com/paper/2606.22917/) (2026)
- [The Hitchhiker's Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semantics](https://mlsanity.com/paper/2607.13077/) (2026)
- [The Free-Recipe Limit: Every Recipe Effect Measures Which Premise of an Idealised Learner Broke](https://mlsanity.com/paper/2609.26160/) (2026)
- [When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning](https://mlsanity.com/paper/2609.33220/) (2026)

## Abstract (arXiv)

The impact of random seeds in fine-tuning large language models (LLMs) has been largely overlooked despite its potential influence on model performance.In this study, we systematically evaluate the effects of random seeds on LLMs using the GLUE and SuperGLUE benchmarks. We analyze the macro-level impact through traditional metrics like accuracy and F1, calculating their mean and variance to quantify performance fluctuations. To capture the micro-level effects, we introduce a novel metric, consistency, measuring the stability of individual predictions across runs. Our experiments reveal significant variance at both macro and micro levels, underscoring the need for careful consideration of random seeds in fine-tuning and evaluation.