# SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors

arXiv: https://arxiv.org/abs/2405.19597 · HTML: https://mlsanity.com/paper/2405.19597/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 26 of 26)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://mlsanity.com/paper/2404.02948/) (2024)
- [Controlling Text-to-Image Diffusion by Orthogonal Finetuning](https://mlsanity.com/paper/2306.07280/) (2023)
- [Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization](https://mlsanity.com/paper/2311.06243/) (2023)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [Batched Low-Rank Adaptation of Foundation Models](https://mlsanity.com/paper/2312.05677/) (2023)
- [Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification](https://mlsanity.com/paper/2302.08909/) (2023)
- [SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstruction](https://mlsanity.com/paper/2308.14604/) (2023)

## Built on by (most cited first, 30 of 58)

- [No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces](https://mlsanity.com/paper/2502.04959/) (2025)
- [RandLoRA: Full-rank parameter-efficient fine-tuning of large models](https://mlsanity.com/paper/2502.00987/) (2025)
- [On the Crucial Role of Initialization for Matrix Factorization](https://mlsanity.com/paper/2410.18965/) (2024)
- [LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation](https://mlsanity.com/paper/2501.16559/) (2025)
- [Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product](https://mlsanity.com/paper/2508.00230/) (2025)
- [RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models](https://mlsanity.com/paper/2505.18877/) (2025)
- [QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation](https://mlsanity.com/paper/2507.04599/) (2025)
- [DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations](https://mlsanity.com/paper/2502.06029/) (2025)
- [Orthogonal Finetuning Made Scalable](https://mlsanity.com/paper/2506.19847/) (2025)
- [LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2410.13618/) (2024)
- [PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection](https://mlsanity.com/paper/2505.20211/) (2025)
- [ESSA: Evolutionary Strategies for Scalable Alignment](https://mlsanity.com/paper/2507.04453/) (2025)
- [PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning](https://mlsanity.com/paper/2409.16722/) (2024)
- [CoLA: Collaborative Low-Rank Adaptation](https://mlsanity.com/paper/2505.15471/) (2025)
- [MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation](https://mlsanity.com/paper/2506.14436/) (2025)
- [PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark](https://mlsanity.com/paper/2511.21285/) (2025)
- [Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation](https://mlsanity.com/paper/2505.11235/) (2025)
- [SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASR](https://mlsanity.com/paper/2509.02830/) (2025)
- [CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning](https://mlsanity.com/paper/2411.17426/) (2024)
- [Understanding Post-Training Structural Changes in Large Language Models](https://mlsanity.com/paper/2509.17866/) (2025)
- [F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning](https://mlsanity.com/paper/2509.23173/) (2025)
- [DiaBlo: Diagonal Blocks Are Sufficient For Finetuning](https://mlsanity.com/paper/2506.03230/) (2025)
- [VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models](https://mlsanity.com/paper/2503.19530/) (2025)
- [Dual Decomposition of Weights and Singular Value Low Rank Adaptation](https://mlsanity.com/paper/2505.14367/) (2025)
- [LoRMA: Low-Rank Multiplicative Adaptation for LLMs](https://mlsanity.com/paper/2506.07621/) (2025)
- [Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models](https://mlsanity.com/paper/2510.21175/) (2025)
- [AdaGradSelect: An adaptive gradient-guided layer selection method for efficient fine-tuning of SLMs](https://mlsanity.com/paper/2512.15764/) (2025)
- [SSVD-O: Parameter-Efficient Fine-Tuning with Structured SVD for Speech Recognition](https://mlsanity.com/paper/2601.12600/) (2026)
- [Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles](https://mlsanity.com/paper/2601.22068/) (2026)
- [GAIN: Multiplicative Modulation for Domain Adaptation](https://mlsanity.com/paper/2604.04516/) (2026)

## Abstract (arXiv)

Popular parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants, freeze pre-trained model weights (W) and inject learnable matrices (Delta W). These (Delta W) matrices are structured for efficient parameterization, often using techniques like low-rank approximations or scaling vectors. However, these methods typically show a performance gap compared to full fine-tuning. Although recent PEFT methods have narrowed this gap, they do so at the cost of additional learnable parameters. We propose SVFT, a simple approach that fundamentally differs from existing methods: the structure imposed on (Delta W) depends on the specific weight matrix (W). Specifically, SVFT updates (W) as a sparse combination of outer products of its singular vectors, training only the coefficients (scales) of these sparse combinations. This approach allows fine-grained control over expressivity through the number of coefficients. Extensive experiments on language and vision benchmarks show that SVFT recovers up to 96% of full fine-tuning performance while training only 0.006 to 0.25% of parameters, outperforming existing methods that only recover up to 85% performance using 0.03 to 0.8% of the trainable parameter budget.