# AdaSVD: Adaptive Singular Value Decomposition for Large Language Models

arXiv: https://arxiv.org/abs/2502.01403 · HTML: https://mlsanity.com/paper/2502.01403/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2025.

## Built on by (most cited first, 15 of 15)

- [Principled Multimodal Representation Learning](https://mlsanity.com/paper/2507.17343/) (2025)
- [ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration](https://mlsanity.com/paper/2505.24357/) (2025)
- [QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models](https://mlsanity.com/paper/2510.16292/) (2025)
- [SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression](https://mlsanity.com/paper/2602.03051/) (2026)
- [ERC-SVD: Error-Controlled SVD for Large Language Model Compression](https://mlsanity.com/paper/2505.20112/) (2025)
- [AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression](https://mlsanity.com/paper/2604.02119/) (2026)
- [COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation](https://mlsanity.com/paper/2507.07580/) (2025)
- [WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models](https://mlsanity.com/paper/2604.02570/) (2026)
- [CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functions](https://mlsanity.com/paper/2510.14262/) (2025)
- [Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs](https://mlsanity.com/paper/2608.08506/) (2026)
- [Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression](https://mlsanity.com/paper/2605.08568/) (2026)
- [LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Models](https://mlsanity.com/paper/2606.00573/) (2026)
- [LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression](https://mlsanity.com/paper/2607.03057/) (2026)
- [Sequential Functional Structured Tucker Compression for Large Language Model Attentions](https://mlsanity.com/paper/2610.00717/) (2026)
- [GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression](https://mlsanity.com/paper/2610.15398/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable success in natural language processing (NLP) tasks, yet their substantial memory requirements present significant challenges for deployment on resource-constrained devices. Singular Value Decomposition (SVD) has emerged as a promising compression technique for LLMs, offering considerable reductions in memory overhead. However, existing SVD-based methods often struggle to effectively mitigate the errors introduced by SVD truncation, leading to a noticeable performance gap when compared to the original models. Furthermore, applying a uniform compression ratio across all transformer layers fails to account for the varying importance of different layers. To address these challenges, we propose AdaSVD, an adaptive SVD-based LLM compression approach. Specifically, AdaSVD introduces adaComp, which adaptively compensates for SVD truncation errors by alternately updating the singular matrices U and V^top. Additionally, AdaSVD introduces adaCR, which adaptively assigns layer-specific compression ratios based on the relative importance of each layer. Extensive experiments across multiple LLM/VLM families and evaluation metrics demonstrate that AdaSVD consistently outperforms state-of-the-art (SOTA) SVD-based methods, achieving superior performance with significantly reduced memory requirements. Code and models of AdaSVD will be available at https://github.com/ZHITENGLI/AdaSVD.