mlsanityData to 7 Oct 2026

Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters

arXiv 2407.16712 ↗First posted 22 Jul 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged, thus, resulting in a highly sparse adapter. This high sparsity incurs no inference overhead, enables rapid switching directly in the fused mode, and significantly reduces concept-loss during multi-adapter fusion. Our extensive experiments on LVMs and LLMs demonstrate that finetuning merely 1-2% parameters in the base model is sufficient for many adapter tasks and significantly outperforms Low Rank Adaptation (LoRA). We also show that SHiRA is orthogonal to advanced LoRA methods such as DoRA and can be easily combined with existing techniques.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 23 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

19 of 23 references.

References · 19

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  4. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  5. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  6. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  7. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  8. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  9. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  10. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  11. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  12. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
7 more
  1. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  2. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  3. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  4. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  5. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  6. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  7. 2024Sparse High Rank Adapterscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  2. 2025SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networkscited by 2
  3. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  4. 2025QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Modelscited by 0
  5. 2026FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptationcited by 0