mlsanityData to 14 Oct 2026

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

arXiv 2410.10054 ↗First posted 14 Oct 2024cs.CLcited by 20

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), are known to enhance training efficiency in Large Language Models (LLMs). Due to the limited parameters of LoRA, recent studies seek to combine LoRA with Mixture-of-Experts (MoE) to boost performance across various tasks. However, inspired by the observed redundancy in traditional MoE structures, previous studies identify similar redundancy among LoRA experts within the MoE architecture, highlighting the necessity for non-uniform allocation of LoRA experts across different layers. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory to design a fine-grained allocation strategy. Our analysis reveals that the number of experts per layer correlates with layer training quality, which exhibits significant variability across layers. Based on this, we introduce AlphaLoRA, a theoretically principled and training-free method for allocating LoRA experts to further mitigate redundancy. Experiments on three models across ten language processing and reasoning benchmarks demonstrate that AlphaLoRA achieves comparable or superior performance over all baselines. Our code is available at https://github.com/morelife2017/alphalora.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 28 of its 49 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

28 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  2. 2024Higher Layers Need More LoRA Experts13 Feb

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2023Mistral 7Bcited by 3,164
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
14 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  5. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  6. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  7. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  8. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  9. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  10. 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 44
  11. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  12. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15
  13. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 12
  14. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 10.

20 papers.

Sort:

By area:

  1. 2025Advancing Expert Specialization for Better MoEcited by 39
  2. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  3. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  4. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 13
  5. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 12
  6. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  7. 2024Model Balancing Helps Low-data Training and Fine-tuningcited by 11
  8. 2025Eigenspectrum Analysis of Neural Networks without Aspect Ratio Biascited by 10
  9. 2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectorscited by 6
  10. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuningcited by 6
  11. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuningcited by 2
  12. 2026Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?cited by 1
8 more
  1. 2026Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuningcited by 1
  2. 2026Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networkscited by 1
  3. 2026Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimizationcited by 0
  4. 2026Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocationcited by 0
  5. 2026Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compressioncited by 0
  6. 2026Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMscited by 0
  7. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  8. 2026What Does Layer-Importance Reveal About Transformers and State-Space Models?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
  2. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
  3. 2025L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
  4. 2026HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  6. 2026DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
  7. 2025MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
  8. 2026TLoRA: Task-aware Low Rank Adaptation of Large Language Models