mlsanityData to 14 Oct 2026

New LoRA Skills Should Read but Never Write

arXiv 2609.31600 ↗First posted 25 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Low-rank adapters (LoRA) make it cheap to fine-tune a large language model once per task, but combining several independently trained adapters into one model remains difficult: merging the updates in weight space causes interference, retraining on all task data is expensive, and routing between separate adapters gives up the goal of a single combined model. We trace the difficulty to two choices that every composition method makes implicitly. A LoRA update admits infinitely many equivalent factorizations; the choice among them is invisible while an adapter serves alone, but it determines what a learned interaction between adapters can see. A coupling between an old skill and a new one can likewise point in either direction, and the direction decides whether the old skills keep computing what they computed before. We introduce READ (Read-only Expansion of Adapter Deltas), which fixes both choices: each adapter is rewritten into a balanced canonical form that preserves its update exactly, and the coupling grows in one direction only, so a new skill can read the input subspaces of old skills but cannot write into their output subspaces. The only trainable object at each append is the new skill's row of the coupling matrix, and the composed update folds into the base weights with no inference cost, routing, or task-specific rules. We evaluate READ across four benchmark suites and two model families, adding skills one at a time. Across several families, READ improves every suite average over the strongest published baselines built from the same adapters---by more than twenty points on SuperGLUE and more than seven points on the domain suite---and nearly all complete addition sequences end above every direct baseline. Factor coordinates and coupling direction, which a lone adapter never exposes, are what decide whether composed skills survive.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 45 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

43 of 45 references.

References · 43

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  12. 2022Editing Models with Task Arithmeticcited by 1,120
31 more
  1. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  8. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  10. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  11. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  12. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  13. 2017Variational Continual Learningcited by 339
  14. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  15. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  16. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  17. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  18. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  19. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  20. 2024Model merging with SVD to tie the Knotscited by 122
  21. 2024EMR-Merging: Tuning-Free High-Performance Model Mergingcited by 110
  22. 2024Model Stock: All we need is just a few fine-tuned modelscited by 92
  23. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  24. 2024Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clusteringcited by 59
  25. 2024What Matters for Model Merging at Scale?cited by 55
  26. 2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 52
  27. 2025Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Mergingcited by 19
  28. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 18
  29. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  30. 2026Label-Free Cross-Task LoRA Merging with Null-Space Compressioncited by 4
  31. 2026Compress then Merge: From Multiple LoRAs into One Low-Rank Adaptercited by 3

Built on by

Not cited yet.