mlsanityData to 7 Oct 2026

FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts

arXiv 2510.08396 ↗First posted 9 Oct 2025cs.LG cs.AI cs.CLcited by 34

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for foundation models, but it suffers from parameter interference, resulting in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA variants show promise in mitigating intra-task correlations in single-task instruction tuning, they introduce additional router parameters and remain ineffective in multi-task model merging where inter-task interference arises. Inspired by the fly olfactory circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix, and (2) an implicit router that unifies expert routing and down-projection, where a frozen sparse random projection matrix replaces the traditional dense trainable version. This design resolves the trade-off between intra-task decorrelation and computational efficiency by eliminating the need for an explicit router, while inherently mitigating inter-task interference due to the orthogonality property of random matrices. Extensive experiments across four domains -- general knowledge understanding, scientific question answering, mathematical reasoning, and code generation -- demonstrate consistent performance improvements over existing methods. Beyond empirical gains, FlyLoRA highlights how biological structures can inspire innovations in AI technologies. Code is available at https://github.com/gfyddha/FlyLoRA.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 75 of its 98 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

75 of 98 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  2. 2024Higher Layers Need More LoRA Experts13 Feb
  3. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin15 Dec

Other references · 72

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  12. 2025Group Sequence Policy Optimizationcited by 788
  13. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  14. 2021GPT Understands, Toocited by 715
  15. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  16. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  17. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  18. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  19. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  20. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  21. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  22. 2021Hash Layers For Large Sparse Modelscited by 236
  23. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  24. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  25. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  26. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  27. 2024Mixture of LoRA Expertscited by 179
  28. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  29. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  30. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  31. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  32. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  33. 2024Model merging with SVD to tie the Knotscited by 118
  34. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  35. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  36. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  37. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  38. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 32.

34 papers.

Sort:

By area:

  1. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 14
  2. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10
  3. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  4. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  5. 2026Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projectioncited by 8
  6. 2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptationcited by 6
  7. 2026CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Mergingcited by 4
  8. 2026NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptationcited by 4
  9. 2026Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projectioncited by 4
  10. 2026Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRAcited by 2
  11. 2026Representation Collapse in Sequential Post-Training of Large Language Modelscited by 2
  12. 2026Predicting Mergeability of Parameter-Efficient Fine-Tuning Updatescited by 2
22 more
  1. 2026Uncertainty Is Not Enough: Value-of-Information Routing for Mixtures of LoRA Expertscited by 2
  2. 2026Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognitioncited by 1
  3. 2026The Long-Term Effects of Data Selection in LLM Fine-Tuningcited by 1
  4. 2026Online Data Selection Is Implicit Alignmentcited by 1
  5. 2026Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRAcited by 1
  6. 2026FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Modelscited by 0
  7. 2026CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategiescited by 0
  8. 2026Early Intervention for VFM-based Multimodal Medical Image Classificationcited by 0
  9. 2026M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skillscited by 0
  10. 2026MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptationcited by 0
  11. 2026Knowledge-Preserved Model Tuning in Null-Space for Robust Spatio-Temporal Video Groundingcited by 0
  12. 2026Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuningcited by 0
  13. 2026Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participationcited by 0
  14. 2026Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuningcited by 0
  15. 2026PERO: Efficient Robust Post-Training Foundation Models for Encrypted Traffic Classificationcited by 0
  16. 2026SPARCL: Spectral Partitioned Analytic Continual Learningcited by 0
  17. 2026LoRA-RC: Reservoir Computing with Low-Rank Adaptationcited by 0
  18. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  19. 2026You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinementcited by 0
  20. 2026ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMscited by 0
  21. 2026Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuningcited by 0
  22. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  2. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
  3. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
  4. 2026MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
  5. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  6. 2026MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
  7. 2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning
  8. 2023SiRA: Sparse Mixture of Low Rank Adaptation