mlsanityData to 14 Oct 2026

MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models

arXiv 2506.05928 ↗First posted 6 Jun 2025cs.CL cs.AIcited by 3

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Recent studies integrate Low-Rank Adaptation (LoRA) and Mixture-of-Experts (MoE) to further enhance the performance of parameter-efficient fine-tuning (PEFT) methods in Large Language Model (LLM) applications. Existing methods employ homogeneous MoE-LoRA architectures composed of LoRA experts with either similar or identical structures and capacities. However, these approaches often suffer from representation collapse and expert load imbalance, which negatively impact the potential of LLMs. To address these challenges, we propose a heterogeneous Mixture-of-Adapters (MoA) approach. This method dynamically integrates PEFT adapter experts with diverse structures, leveraging their complementary representational capabilities to foster expert specialization, thereby enhancing the effective transfer of pre-trained knowledge to downstream tasks. MoA supports two variants: (i) textitSoft MoA achieves fine-grained integration by performing a weighted fusion of all expert outputs; (ii) textitSparse MoA activates adapter experts sparsely based on their contribution, achieving this with negligible performance degradation. Experimental results demonstrate that heterogeneous MoA outperforms homogeneous MoE-LoRA methods in both performance and parameter efficiency. Our project is available at https://github.com/DCDmllm/MoA.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 30 of its 42 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

30 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Higher Layers Need More LoRA Experts13 Feb
  2. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin15 Dec
  3. 2023SiRA: Sparse Mixture of Low Rank Adaptation15 Nov

Other references · 27

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
15 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2016Solving General Arithmetic Word Problemscited by 360
  6. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  7. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  8. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  9. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  10. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  11. 2024A Closer Look at the Limitations of Instruction Tuningcited by 77
  12. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  13. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 34
  14. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 29
  15. 2022AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Modelscited by 0
  2. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMscited by 0
  3. 2026Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
  2. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
  3. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
  4. 2025L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
  5. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
  6. 2024T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning
  7. 2025MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
  8. 2025MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models