mlsanityData to 13 Oct 2026

Rethinking Learning Rate Tuning in the Era of Large Language Models

arXiv 2309.08859 ↗First posted 16 Sep 2023cs.LGcited by 16

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models (LLMs) represent the recent success of deep learning in achieving remarkable human-like predictive performance. It has become a mainstream strategy to leverage fine-tuning to adapt LLMs for various real-world applications due to the prohibitive expenses associated with LLM training. The learning rate is one of the most important hyperparameters in LLM fine-tuning with direct impacts on both fine-tuning efficiency and fine-tuned LLM quality. Existing learning rate policies are primarily designed for training traditional deep neural networks (DNNs), which may not work well for LLM fine-tuning. We reassess the research challenges and opportunities of learning rate tuning in the coming era of Large Language Models. This paper makes three original contributions. First, we revisit existing learning rate policies to analyze the critical challenges of learning rate tuning in the era of LLMs. Second, we present LRBench++ to benchmark learning rate policies and facilitate learning rate tuning for both traditional DNNs and LLMs. Third, our experimental analysis with LRBench++ demonstrates the key differences between LLM fine-tuning and traditional DNN training and validates our analysis.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 35 of its 53 references that are in the map. First posted on arXiv 16 Sep 2023.

Builds on

35 of 53 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
23 more
  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,472
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,496
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,488
  7. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,439
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  9. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 693
  10. 2018CoQA: A Conversational Question Answering Challengecited by 681
  11. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  12. 2018Object Detection with Deep Learning: A Reviewcited by 527
  13. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  14. 2017Population Based Training of Neural Networkscited by 359
  15. 2019Self-Driving Cars: A Surveycited by 148
  16. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  17. 2021AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Flycited by 14
  18. 2019Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networkscited by 12
  19. 2022Selecting and Composing Learning Rate Policies for Deep Neural Networkscited by 11
  20. 2018A Comparative Measurement Study of Deep Learning as a Service Frameworkcited by 10
  21. 2021Parallel Detection for Efficient Video Analytics at the Edgecited by 6
  22. 2020AutoLR: An Evolutionary Approach to Learning Rate Policiescited by 4
  23. 2023Fast and Resource-Efficient Object Tracking on Edge Devices: A Measurement Studycited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 185
  2. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretrainingcited by 31
  3. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  4. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  5. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 22
  6. 2025Mid-Training of Large Language Models: A Surveycited by 19
  7. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  8. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 13
  9. 2025Digital Forensics in the Age of Large Language Modelscited by 10
  10. 2026Loop the Loopies!cited by 7
  11. 2025Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applicationscited by 5
  12. 2024Designing Domain-Specific Large Language Models: The Critical Role of Fine-Tuning in Public Opinion Simulationcited by 4
4 more
  1. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  2. 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agentscited by 2
  3. 2025FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Modelcited by 1
  4. 2025Multi-Model Synthetic Training for Mission-Critical Small Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  2. 2023Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models
  3. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
  4. 2024A critical review of methods and challenges in large language models
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Models
  6. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
  7. 2024TAIA: Large Language Models are Out-of-Distribution Data Learners
  8. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning