mlsanityData to 7 Oct 2026

Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment

arXiv 2502.16894 ↗First posted 24 Feb 2025cs.CLcited by 22

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for Large Language Models (LLMs), its performance often falls short of Full Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with static singular value decomposition (SVD) subsets, leading to suboptimal leveraging of pre-trained knowledge. Another path for improving LoRA is incorporating a Mixture-of-Experts (MoE) architecture. However, weight misalignment and complex gradient dynamics make it challenging to adopt SVD prior to the LoRA MoE architecture. To mitigate these issues, we propose underlineGreat LunderlineoRunderlineA Mixture-of-Experunderlinet (GOAT), a framework that (1) adaptively integrates relevant priors using an SVD-structured MoE, and (2) aligns optimization with full fine-tuned MoE by deriving a theoretical scaling factor. We demonstrate that proper scaling, without modifying the architecture or training algorithms, boosts LoRA MoE's efficiency and performance. Experiments across 25 datasets, including natural language understanding, commonsense reasoning, image classification, and natural language generation, demonstrate GOAT's state-of-the-art performance, closing the gap with Full FT.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 46 of its 84 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

46 of 84 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
34 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  12. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  13. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  14. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  15. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  16. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  17. 2024LoRA Learns Less and Forgets Lesscited by 313
  18. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  19. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  20. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  21. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  22. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  23. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  24. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  25. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 114
  26. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  27. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  28. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 33
  29. 2023MultiLoRA: Democratizing LoRA for Better Multi-Task Learningcited by 32
  30. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 25
  31. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  32. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  33. 2024On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusioncited by 9
  34. 2023Enhancing Low-Resource Relation Representations through Multi-View Decouplingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 16.

22 papers.

Sort:

By area:

  1. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  2. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  3. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  4. 2026W2T: LoRA Weights Already Know What They Can Docited by 4
  5. 2026FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuningcited by 3
  6. 2026Exploring Data-Free LoRA Transferability for Video Diffusion Modelscited by 3
  7. 2025C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Modelscited by 2
  8. 2025STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Frameworkcited by 2
  9. 2025Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Modelscited by 2
  10. 2025ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimationcited by 2
  11. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  12. 2026Balanced LoRA: Removing Parameter Invariance to Accelerate Convergencecited by 2
10 more
  1. 2026LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modulescited by 1
  2. 2026Sparse Spectral LoRA: Routed Experts for Medical VLMscited by 1
  3. 2026Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuningcited by 1
  4. 2026VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Expertscited by 1
  5. 2025FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergencecited by 0
  6. 2026FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Modelscited by 0
  7. 2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
  8. 2026Prompt2Effect: Training-Free Image-to-Video Model Specialization via LoRA Generationcited by 0
  9. 2026Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAscited by 0
  10. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  2. 2025LoRA Is Slower Than You Think
  3. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  4. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality
  5. 2024Higher Layers Need More LoRA Experts
  6. 2025Uni-LoRA: One Vector is All You Need
  7. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptation
  8. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling