mlsanityData to 14 Oct 2026

LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing

arXiv 2507.00029 ↗First posted 17 Jun 2025cs.LG cs.AIcited by 18

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Recent attempts to combine low-rank adaptation (LoRA) with mixture-of-experts (MoE) for multi-task adaptation of Large Language Models (LLMs) often replace whole attention/FFN layers with switch experts or append parallel expert branches, undermining parameter efficiency and limiting task specialization. We introduce LoRA-Mixer, a modular MoE framework that routes task-specific LoRA experts into the core projection matrices of the attention module, namely input and output linear layers, rather than primarily targeting FFN blocks. The design delivers fine-grained token-level specialization by fully exploiting the attention mechanism, while remaining drop-in compatible with Transformers and state-space models (SSMs), since linear projection layers are ubiquitous. To train robust routers from limited data while promoting stable, selective decisions and high expert reuse, LoRA-Mixer employs an adaptive Routing Specialization Loss (RSL) that jointly enforces global load balance and input-aware specialization via an entropy-shaping objective. The framework supports two regimes: (i) joint optimization of adapters and router with a differentiable hard-soft top-k routing scheme, and (ii) plug-and-play routing over frozen, pre-trained LoRA modules sourced from public repositories. Across 15 benchmarks, including MedQA, GSM8K, HumanEval, and GLUE, RSL-optimized LoRA-Mixer outperforms state-of-the-art routing and LoRA-MoE baselines while using 48 percent of their trainable parameters, with gains of 3.79, 2.90, and 3.95 percentage points on GSM8K, CoLA, and ARC-C, respectively. Cross-model transfer and adapter reuse experiments further demonstrate the approach's versatility and data efficiency. Our code is available at https://github.com/hustcselwb/LoRA-Mixer.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 58 of its 74 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

58 of 74 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism1 Apr
  2. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment24 Feb
  3. 2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models19 May
  4. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  5. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models20 Feb
  6. 2024Higher Layers Need More LoRA Experts13 Feb
  7. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  8. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin15 Dec

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  5. 2021GPT Understands, Toocited by 716
  6. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  7. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  8. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  9. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  10. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  11. 2024LoRA Learns Less and Forgets Lesscited by 328
  12. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  13. 2024Mixture of LoRA Expertscited by 181
  14. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  15. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  16. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  17. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  18. 2022Transformer Tracking with Cyclic Shifting Window Attentioncited by 75
  19. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 74
  20. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  21. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  22. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  23. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  24. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 59
  25. 2023Compact Transformer Tracker with Correlative Masked Modelingcited by 50
  26. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  27. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 46
  28. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 42
  29. 2025Advancing Expert Specialization for Better MoEcited by 39
  30. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  31. 2025Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Modelscited by 32
  32. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 31
  33. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  34. 2025SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understandingcited by 25
  35. 2025GA-S3: Comprehensive Social Network Simulation with Group Agentscited by 19
  36. 2025LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selectioncited by 14
  37. 2025H-MoRe: Learning Human-centric Motion Representation for Action Analysiscited by 8
  38. 2024GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaborationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area:

  1. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 25
  2. 2026Seeing Further and Wider: Joint Spatio-Temporal Enlargement for Micro-Video Popularity Predictioncited by 5
  3. 2026Large Language Model as Token Compressor and Decompressorcited by 4
  4. 2026HotComment: A Benchmark for Evaluating Popularity of Online Commentscited by 3
  5. 2026CurEvo: Curriculum-Guided Self-Evolution for Video Understandingcited by 3
  6. 2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winnercited by 2
  7. 2026Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognitioncited by 1
  8. 2026DiFA: Inference-Time Forward-Process Alignment for Diffusion Modelscited by 1
  9. 2026Semiotic logical hexagon theory for LLM logical reasoningcited by 1
  10. 2026GateMOT: Q-Gated Attention for Dense Object Trackingcited by 0
  11. 2026PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specializationcited by 0
  12. 2026Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAscited by 0
6 more
  1. 2026Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answeringcited by 0
  2. 2026Behavior Quotient Learning for Low-Rank Adaptation of LLM Agentscited by 0
  3. 2026Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagementcited by 0
  4. 2026New LoRA Skills Should Read but Never Writecited by 0
  5. 2026Conditional Rank Allocation for Taxonomy-Aware Medical Language Model Adaptationcited by 0
  6. 2026How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
  2. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
  3. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  4. 2025L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
  5. 2026SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
  6. 2025From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning
  7. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
  8. 2025MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation