mlsanityData to 7 Oct 2026

Mixture-of-Subspaces in Low-Rank Adaptation

arXiv 2406.11909 ↗First posted 16 Jun 2024cs.LG cs.AIcited by 47

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In this paper, we introduce a subspace-inspired Low-Rank Adaptation (LoRA) method, which is computationally efficient, easy to implement, and readily applicable to large language, multimodal, and diffusion models. Initially, we equivalently decompose the weights of LoRA into two subspaces, and find that simply mixing them can enhance performance. To study such a phenomenon, we revisit it through a fine-grained subspace lens, showing that such modification is equivalent to employing a fixed mixer to fuse the subspaces. To be more flexible, we jointly learn the mixer with the original LoRA weights, and term the method Mixture-of-Subspaces LoRA (MoSLoRA). MoSLoRA consistently outperforms LoRA on tasks in different modalities, including commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation, demonstrating its effectiveness and robustness. Codes are available at https://github.com/wutaiqiang/MoSLoRA.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 45 of its 63 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

45 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation26 Sep
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
31 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  13. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  15. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  16. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  17. 2016A Diagram Is Worth A Dozen Imagescited by 793
  18. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  19. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  20. 2021GPT Understands, Toocited by 715
  21. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  22. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  23. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  24. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  25. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  26. 2020ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgettingcited by 85
  27. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  28. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  29. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  30. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  31. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 25; 2026: 13.

47 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  3. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  4. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  5. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  6. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  7. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  8. 2024Flexora: Flexible Low Rank Adaptation for Large Language Modelscited by 13
  9. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  10. 2024Low-Rank Interconnected Adaptation across Layerscited by 10
  11. 2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
  12. 2025Federated Continual Instruction Tuningcited by 10
35 more
  1. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  2. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
  3. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
  4. 2025Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extractioncited by 7
  5. 2025Shadow-FT: Tuning Instruct Model via Training on Paired Base Modelcited by 7
  6. 2025Orthogonal Finetuning Made Scalablecited by 7
  7. 2025PaCA: Partial Connection Adaptation for Efficient Fine-Tuningcited by 6
  8. 2025Weight Spectra Induced Efficient Model Adaptationcited by 5
  9. 2024LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Modelscited by 4
  10. 2025DenseLoRA: Dense Low-Rank Adaptation of Large Language Modelscited by 4
  11. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  12. 2025S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuningcited by 3
  13. 2025One Rank at a Time: Cascading Error Dynamics in Sequential Learningcited by 3
  14. 2025AgeBooth: Controllable Facial Aging and Rejuvenation via Diffusion Modelscited by 3
  15. 2026FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptationcited by 3
  16. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  17. 2025Predicting Through Generation: Why Generation Is Better for Predictioncited by 2
  18. 2025HaLoRA: Hardware-aware Low-Rank Adaptation for Large Language Models Based on Hybrid Compute-in-Memory Architecturecited by 2
  19. 2025DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuningcited by 2
  20. 2025SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networkscited by 2
  21. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
  22. 2025MAP: Revisiting Weight Decomposition for Low-Rank Adaptationcited by 1
  23. 2026Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Expertscited by 1
  24. 2024MoR: Mixture of Ranks for Low-Rank Adaptation Tuningcited by 0
  25. 2024Linear Chain Transformation: Expanding Optimization Dynamics for Fine-Tuning Large Language Modelscited by 0
  26. 2025Expanding the Content-Style Frontier: a Balanced Subspace Blending Approach for Content-Style LoRA Fusioncited by 0
  27. 2026Manifold-Aware Temporal Domain Generalization for Large Language Modelscited by 0
  28. 2026Dynamic Training-Free Fusion of Subject and Style LoRAscited by 0
  29. 2026NOBLE: Accelerating Transformers with Nonlinear Low-Rank Branchescited by 0
  30. 2026SOLAR: Communication-Efficient Model Adaptation via Subspace-Oriented Latent Adapter Reparametrizationcited by 0
  31. 2026SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptationcited by 0
  32. 2026Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuningcited by 0
  33. 2026DeCo: Zero-Shot Industrial Anomaly Generation through Decoupling and Recouplingcited by 0
  34. 2026Normalized Low-Rank Adaptationcited by 0
  35. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
  2. 2025Uni-LoRA: One Vector is All You Need
  3. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  4. 2025LoRAtorio: An intrinsic approach to LoRA Skill Composition
  5. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  6. 2026Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
  7. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  8. 2025One Head Eight Arms: Block Matrix based Low Rank Adaptation for CLIP-based Few-Shot Learning