mlsanityData to 14 Oct 2026

Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models

arXiv 2503.07329 ↗First posted 10 Mar 2025cs.CL cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

The impact of random seeds in fine-tuning large language models (LLMs) has been largely overlooked despite its potential influence on model performance.In this study, we systematically evaluate the effects of random seeds on LLMs using the GLUE and SuperGLUE benchmarks. We analyze the macro-level impact through traditional metrics like accuracy and F1, calculating their mean and variance to quantify performance fluctuations. To capture the micro-level effects, we introduce a novel metric, consistency, measuring the stability of individual predictions across runs. Our experiments reveal significant variance at both macro and micro levels, underscoring the need for careful consideration of random seeds in fine-tuning and evaluation.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 14 of its 31 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

14 of 31 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  6. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  7. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  8. 2021Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer visioncited by 81
  9. 2023Coverage-based Example Selection for In-Context Learningcited by 59
  10. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  11. 2019On Model Stability as a Function of Random Seedcited by 31
  12. 2023On The Impact of Machine Learning Randomness on Group Fairnesscited by 26
2 more
  1. 2022We need to talk about random seedscited by 13
  2. 2020How many images do I need? Understanding how sample size per class affects deep learning model performance metrics for balanced designs in autonomous wildlife monitoringcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2026The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?cited by 10
  2. 2025GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Librariescited by 8
  3. 2026Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Modelscited by 3
  4. 2026On the Extreme Variance of Certified Local Robustness Across Model Seedscited by 0
  5. 2026Diagnostics for Individual-Level Prediction Instability in Machine Learning for Healthcarecited by 0
  6. 2026Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classificationscited by 0
  7. 2026GRAIN: Group Aggregation via Min-Norm Objectivecited by 0
  8. 2026The Hitchhiker's Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semanticscited by 0
  9. 2026The Free-Recipe Limit: Every Recipe Effect Measures Which Premise of an Idealised Learner Brokecited by 0
  10. 2026When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learningcited by 0