mlsanityData to 7 Oct 2026

RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation

arXiv 2401.04679 ↗First posted 9 Jan 2024cs.CL cs.AI cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

We investigate parameter-efficient fine-tuning (PEFT) methods that can provide good accuracy under limited computational and memory budgets in the context of large language models (LLMs). We present a new PEFT method called Robust Adaptation (RoSA) inspired by robust principal component analysis that jointly trains textitlow-rank and textithighly-sparse components on top of a set of fixed pretrained weights to efficiently approximate the performance of a full-fine-tuning (FFT) solution. Across a series of challenging generative tasks such as grade-school math and SQL query generation, which require fine-tuning for good performance, we show that RoSA outperforms LoRA, pure sparse fine-tuning, and alternative hybrid methods at the same parameter budget, and can even recover the performance of FFT on some tasks. We provide system support for RoSA to complement the training algorithm, specifically in the form of sparse GPU kernels which enable memory- and computationally-efficient training, and show that it is also compatible with low-precision base weights, resulting in the first joint representation combining quantization, low-rank and sparse approximations. Our code is available at https://github.com/IST-DASLab/RoSA.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 52 of its 77 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

52 of 77 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2021GPT Understands, Toocited by 715
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  15. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  16. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  17. 2019The State of Sparsity in Deep Neural Networkscited by 471
  18. 2021MetaICL: Learning to Learn In Contextcited by 450
  19. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  20. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  21. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  22. 2021Accelerating Sparse Deep Neural Networkscited by 213
  23. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  24. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  25. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  26. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  27. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  28. 2020Sparse GPU Kernels for Deep Learningcited by 138
  29. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
  30. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  31. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  32. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  33. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  34. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  35. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 34
  36. 2019ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversationcited by 31
  37. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
  38. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 15; 2026: 9.

48 papers.

Sort:

By area:

  1. 2024LoRA Learns Less and Forgets Lesscited by 313
  2. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  3. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  4. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  5. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  6. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 18
  7. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  8. 2024Sparse High Rank Adapterscited by 17
  9. 2024Parameter Efficient Fine-tuning via Explained Variance Adaptationcited by 17
  10. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
  11. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  12. 2025HALO: Hadamard-Assisted Lower-Precision Optimization for LLMscited by 14
36 more
  1. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  2. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  3. 2025Parameter-Efficient Continual Fine-Tuning: A Surveycited by 11
  4. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 9
  5. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  6. 2024The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Informationcited by 8
  7. 2024GaLore+: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projectioncited by 8
  8. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  9. 2024Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorizationcited by 6
  10. 2024Panza: Design and Analysis of a Fully-Local Personalized Text Writing Assistantcited by 5
  11. 2024Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapterscited by 5
  12. 2024Parameter-Efficient Fine-Tuning via Circular Convolutioncited by 4
  13. 2024FANAL -- Financial Activity News Alerting Language Modeling Frameworkcited by 4
  14. 2025Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Modelscited by 3
  15. 2024A Bayesian Interpretation of Adaptive Low-Rank Adaptationcited by 2
  16. 2024RECAST: Reparameterized, Compact weight Adaptation for Sequential Taskscited by 2
  17. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helpscited by 2
  18. 2025ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraintscited by 2
  19. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  20. 2025Position: It's Time to Act on the Risk of Efficient Personalized Text Generationcited by 1
  21. 2025An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruningcited by 1
  22. 2025GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameterscited by 1
  23. 2026Live Interactive Training for Video Segmentationcited by 1
  24. 2026Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuningcited by 1
  25. 2024CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Modelscited by 0
  26. 2024PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuningcited by 0
  27. 2024OP-LoRA: The Blessing of Dimensionalitycited by 0
  28. 2025Symbiosis: Multi-Adapter Inference and Fine-Tuningcited by 0
  29. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  30. 2025Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptationcited by 0
  31. 2025Random Initialization of Gated Sparse Adapterscited by 0
  32. 2026Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compressioncited by 0
  33. 2026Active Learning with Low-Rank Structure for Data Selectioncited by 0
  34. 2026LoRSA: Toward Generalizable Parameter-Efficient Fine-Tuning for Biomedical Downstream Taskscited by 0
  35. 2026Awaken, Then Scale: Tiny Adaptation for MRI Reconstructioncited by 0
  36. 2026SQUARE: Structured Quantum Representation Adapters as Compact Quadratic Feature Maps for Frozen Language Modelscited by 0