mlsanityData to 14 Oct 2026

Decoupled Relative Learning Rate Schedules

arXiv 2507.03526 ↗First posted 4 Jul 2025cs.LGcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

In this work, we introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models. Traditional methods often apply a uniform learning rate across all network layers, potentially overlooking the unique dynamics of each part. Remarkably, our introduced relative learning rates, RLRS, method accelerates the training process by up to 23%, particularly in complex models such as Mixture of Experts (MoE). Hyperparameters of RLRS can be efficiently tuned on smaller models and then effectively reused on models up to 27× larger. This simple and effective method results in a substantial reduction in training time and computational resources, offering a practical and scalable solution for optimizing large-scale neural networks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 17 of its 21 references that are in the map. First posted on arXiv 4 Jul 2025.

Builds on

17 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 16

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  6. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  8. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  10. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  11. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
4 more
  1. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  2. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  3. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  4. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 27
  2. 2026Deep-layer limit and stability analysis of the basic forward-backward-splitting induced network (II): learning problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficiently Distilling LLMs for Edge Applications
  2. 2026Learning Rate Transfer for Hybrid Transformer-SSM Architectures
  3. 2023WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models
  4. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
  5. 2024Dynamic Layer Tying for Parameter-Efficient Transformers
  6. 2026Rethinking Neural Network Learning Rates: A Stackelberg Perspective
  7. 2024RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization
  8. 2025LLM Inference Acceleration via Efficient Operation Fusion