mlsanityData to 7 Oct 2026

Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA

arXiv 2602.09492 ↗First posted 10 Feb 2026cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Low-rank adaptation (LoRA) is a standard approach for fine-tuning large language models, yet its many variants report conflicting empirical gains, often on the same benchmarks. We show that these contradictions arise from a single overlooked factor: the batch size. When properly tuned, vanilla LoRA often matches the performance of more complex variants. We further propose a proxy-based, cost-efficient strategy for batch size tuning, revealing the impact of rank, dataset size, and model capacity on the optimal batch size. Our findings elevate batch size from a minor implementation detail to a first-order design parameter, reconciling prior inconsistencies and enabling more reliable evaluations of LoRA variants.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

17 of 24 references.

References · 17

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  4. 2024LoRA Learns Less and Forgets Lesscited by 313
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  6. 2018An Empirical Model of Large-Batch Trainingcited by 268
  7. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  8. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  9. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  10. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  11. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  12. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
5 more
  1. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  2. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  3. 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
  4. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  5. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloadscited by 1
  2. 2026Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuningcited by 0
  3. 2026The Hidden Power of Scaling Factor in LoRA Optimizationcited by 0