mlsanityData to 7 Oct 2026

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

arXiv 2404.02948 ↗First posted 3 Apr 2024cs.LG cs.AIcited by 261

Not on the map. Why?

Abstract (arXiv)

To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes Delta W ∈ ℝm × n through the product of two matrices A ∈ ℝm × r and B ∈ ℝr × n, where r ll min(m, n), A is initialized with Gaussian noise, and B with zeros. LoRA freezes the original model W and updates the "Noise & Zero" adapter, which may lead to slow convergence. To overcome this limitation, we introduce Principal Singular values and Singular vectors Adaptation (PiSSA). PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices A and B with the principal components of the original matrix W, and put the remaining components into a residual matrix Wres ∈ ℝm × n which is frozen during fine-tuning. Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance. Comparative experiments of PiSSA and LoRA across 12 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently outperforms LoRA under identical experimental setups. On the GSM8K benchmark, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, surpassing LoRA's 67.7% by 5.16%. Due to the same architecture, PiSSA is also compatible with quantization to further reduce the memory requirement of fine-tuning. Compared to QLoRA, QPiSSA exhibits smaller quantization errors in the initial stages. Fine-tuning LLaMA-3-70B on GSM8K, QPiSSA attains an accuracy of 86.05%, exceeding the performances of QLoRA at 81.73%. Leveraging a fast SVD technique, PiSSA can be initialized in only a few seconds, presenting a negligible cost for transitioning from LoRA to PiSSA. Code is available at https://github.com/GraphPKU/PiSSA.

Key facts. Cited by 261 papers in the map (data through 7 Oct 2026). Builds on 61 of its 73 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

61 of 73 references.

References · 61

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Mistral 7Bcited by 3,065
  8. 2024Mixtral of Expertscited by 1,785
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  14. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  15. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  16. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  17. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  18. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  19. 2024Yi: Open Foundation Models by 01.AIcited by 706
  20. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  21. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  22. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  23. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  24. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  25. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  26. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  27. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  28. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  29. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  30. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  31. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  32. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  33. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  34. 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
  35. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  36. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  37. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  38. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 47; 2025: 124; 2026: 90.

261 papers.

Sort:

By area:

  1. 2024LoRA Learns Less and Forgets Lesscited by 313
  2. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  3. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  4. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
  6. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  7. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  8. 2024A Survey on LoRA of Large Language Modelscited by 67
  9. 2024SVFT: Parameter-Efficient Fine-Tuning with Singular Vectorscited by 57
  10. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  11. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  12. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 52
38 more
  1. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  2. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  3. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  4. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  5. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  6. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 34
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2025RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustnesscited by 25
  9. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  10. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  11. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 23
  12. 2024SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learningcited by 22
  13. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  14. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  15. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 21
  16. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  17. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  18. 2025Principled Multimodal Representation Learningcited by 19
  19. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 18
  20. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  21. 2024Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptationcited by 15
  22. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  23. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  24. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  25. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15
  26. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  27. 2024SLIP: Securing LLMs IP Using Weights Decompositioncited by 13
  28. 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
  29. 2025Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorizationcited by 13
  30. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
  31. 2025LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapterscited by 13
  32. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  33. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptationcited by 11
  34. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  35. 2025FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasetscited by 11
  36. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  37. 2024Low-Rank Interconnected Adaptation across Layerscited by 10
  38. 2024SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Valuescited by 10

50 most cited of 261 All 261 (JSON)