mlsanityData to 14 Oct 2026

SiRA: Sparse Mixture of Low Rank Adaptation

arXiv 2311.09179 ↗First posted 15 Nov 2023cs.CLcited by 32

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Parameter Efficient Tuning has been an prominent approach to adapt the Large Language Model to downstream tasks. Most previous works considers adding the dense trainable parameters, where all parameters are used to adapt certain task. We found this less effective empirically using the example of LoRA that introducing more trainable parameters does not help. Motivated by this we investigate the importance of leveraging "sparse" computation and propose SiRA: sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top k experts routing with a capacity limit restricting the maximum number of tokens each expert can process. We propose a novel and simple expert dropout on top of gating network to reduce the over-fitting issue. Through extensive experiments, we verify SiRA performs better than LoRA and other mixture of expert approaches across different single tasks and multitask settings.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 30 of its 40 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

30 of 40 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
18 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  5. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  6. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  8. 2023From Sparse to Soft Mixtures of Expertscited by 248
  9. 2021Hash Layers For Large Sparse Modelscited by 237
  10. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  11. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  12. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  13. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  14. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  15. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  16. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  17. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  18. 2022Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 18; 2025: 6; 2026: 7.

32 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  2. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  3. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  4. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  5. 2024A Survey on LoRA of Large Language Modelscited by 67
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  7. 2025Advancing Expert Specialization for Better MoEcited by 39
  8. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 34
  9. 2026Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuningcited by 17
  10. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  11. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 13
  12. 2024Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptationcited by 10
20 more
  1. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  2. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  3. 2024PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learningcited by 8
  4. 2024Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inferencecited by 7
  5. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  6. 2024MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuningcited by 6
  7. 2024MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Expertscited by 6
  8. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Expertscited by 6
  9. 2024MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learningcited by 5
  10. 2024MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learningcited by 4
  11. 2025MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 3
  12. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  13. 2024BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-Task Large Language Modelscited by 1
  14. 2024Collaborative and Efficient Personalization with Mixtures of Adaptorscited by 1
  15. 2026Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learningcited by 1
  16. 2024Inducing Generalization across Languages and Tasks using Featurized Low-Rank Mixturescited by 0
  17. 2026Improving Ensemble Forecasts of Abnormally Deflecting Tropical Cyclones with Fused Atmosphere-Ocean-Terrain Datacited by 0
  18. 2026Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptationcited by 0
  19. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMscited by 0
  20. 2026Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
  2. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  3. 2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
  4. 2026SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
  5. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
  6. 2025L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts
  7. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
  8. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts