mlsanityData to 7 Oct 2026

RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism

arXiv 2605.25565 ↗First posted 25 May 2026cs.LG cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

While Large Language Models (LLMs) are commonly fine-tuned to handle domain-specific tasks before being applied to vertical applications, adapting them to complex scenarios with diverse specialized knowledge remains challenging. Meanwhile, Mixture-of-Experts (MoE) architecture has risen as a crucial paradigm for training LLMs, and some recent works have also incorporated MoE into Parameter-Efficient Fine-Tuning (PEFT) to propose the Mixture of Low-rank Experts (MoE-LoRA), to enhance the power of low-rank adapters for learning complicated knowledge. However, conventional gating mechanisms in MoE typically apply only a scalar reweighing to selected experts, thereby limiting their underlying capacity of representation and generalization. Motivated and enabled by the low-rank structures in MoE-LoRA, we propose RotMoLE, a specialized MoE framework for low-rank experts featuring an additional rotation gate. Beyond simple scaling, RotMoLE implements a rotation mechanism for each selected expert, enabling superior expert exploitation and specialization for learning diverse data, especially when expert candidates are limited. Empirical results on complex multi-task and multilingual training scenarios validate our effectiveness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 57 of its 66 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

57 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Mixture of Experts in Large Language Models15 Jul
  2. 2024Qwen2.5 Technical Report19 Dec
  3. 2024A Survey on Mixture of Experts in Large Language Models26 Jun
  4. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning17 Feb
  5. 2024Mixtral of Experts8 Jan
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 50

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
38 more
  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  5. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  6. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  7. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  8. 2024Mixture of LoRA Expertscited by 179
  9. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  10. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  11. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  12. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  13. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  14. 2024Higher Layers Need More LoRA Expertscited by 101
  15. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  16. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  17. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  18. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  19. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
  20. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  21. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  22. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  23. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 33
  24. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  25. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  26. 2021MTG: A Benchmark Suite for Multilingual Text Generationcited by 22
  27. 2024Yuan 2.0-M32: Mixture of Experts with Attention Routercited by 18
  28. 2024LoRA Meets Dropout under a Unified Frameworkcited by 17
  29. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  30. 2025Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approachcited by 8
  31. 2026CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Mergingcited by 4
  32. 2025Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Modelscited by 3
  33. 2025Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuningcited by 3
  34. 2026MoLoRA: Composable Specialization via Per-Token Adapter Routingcited by 3
  35. 2026SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learningcited by 2
  36. 2024BIG-MoE: Bypass Isolated Gating MoE for Generalized Multimodal Face Anti-Spoofingcited by 1
  37. 2026ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuningcited by 1
  38. 2026TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
  2. 2025Advancing Expert Specialization for Better MoE
  3. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
  4. 2023Adaptive Gating in Mixture-of-Experts based Language Models
  5. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  6. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
  7. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  8. 2026FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models