mlsanityData to 14 Oct 2026

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

arXiv 2405.17604 ↗First posted 27 May 2024cs.LG cs.AI cs.CLcited by 76

Not on the map. Why?

Abstract (arXiv)

The growth of large language models underscores the need for parameter-efficient fine-tuning. Despite its popularity, LoRA encounters storage and computational challenges when deploying multiple task- or user-specific modules. To address this, we introduce LoRA-XS, a novel fine-tuning method backed by a theoretical derivation. LoRA-XS drastically reduces trainable parameters by incorporating a small, trainable weight matrix between frozen low-rank matrices derived from the Singular Value Decomposition of pre-trained weights. This design enables LoRA-XS to reduce storage requirements by over 100x in 7B models compared to LoRA. Additionally, unlike other methods, LoRA-XS imposes no lower bound on trainable parameters - it can scale from a single parameter per module to arbitrarily large values, adapting to any storage or computational constraint. Evaluations on GLUE, GSM8K, MATH, and commonsense reasoning benchmarks across different model scales reveal that LoRA-XS consistently outperforms or matches LoRA and VeRA in accuracy, offering unmatched parameter efficiency. Our ablation studies highlight the significance of singular vectors in transformer weights, establishing LoRA-XS as a powerful, storage-efficient solution for scaling and personalizing large language models.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

33 of 44 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
21 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2023Mistral 7Bcited by 3,164
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  12. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  13. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  14. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  15. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  16. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  17. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  18. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  19. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  20. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  21. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 36; 2026: 30.

76 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  3. 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 41
  4. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 35
  5. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 22
  6. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 20
  7. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  8. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 14
  9. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 14
  10. 2026Learning to Reason in 13 Parameterscited by 14
  11. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  12. 2025LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptationcited by 11
38 more
  1. 2024CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigationcited by 10
  2. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  3. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  4. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuningcited by 10
  5. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  6. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
  7. 2025Layer-wise dynamic rank for compressing large language modelscited by 8
  8. 2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
  9. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  10. 2025Uni-LoRA: One Vector is All You Needcited by 7
  11. 2025The Primacy of Magnitude in Low-Rank Adaptationcited by 7
  12. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 6
  13. 2024PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuningcited by 5
  14. 2025Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRAcited by 5
  15. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4
  16. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  17. 2025EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decompositioncited by 3
  18. 2025ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Modelscited by 3
  19. 2025Mix-of-Language-Experts Architecture for Multilingual Programmingcited by 3
  20. 2025TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptationcited by 3
  21. 2025Taming LLMs by Scaling Learning Rates with Gradient Groupingcited by 2
  22. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helpscited by 2
  23. 2025ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraintscited by 2
  24. 2025SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networkscited by 2
  25. 2026Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsitycited by 2
  26. 2025HRP: High-Rank Preheating for Superior LoRA Initializationcited by 1
  27. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  28. 2025Dual Decomposition of Weights and Singular Value Low Rank Adaptationcited by 1
  29. 2025WaRA: Wavelet Low-Rank Adaptation for Medical Image Classificationcited by 1
  30. 2025QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Modelscited by 1
  31. 2026Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languagescited by 1
  32. 2026LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modulescited by 1
  33. 2026LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weightscited by 1
  34. 2026Spectral Rewiring for Exploration, Purification, and Model Mergingcited by 1
  35. 2026Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewardscited by 1
  36. 2026Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Modelscited by 1
  37. 2026Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?cited by 1
  38. 2026Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptationcited by 1

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LoRA Is Slower Than You Think
  2. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  3. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
  4. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  5. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  6. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  7. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
  8. 2025LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently