# Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

arXiv: https://arxiv.org/abs/2502.02723 · HTML: https://mlsanity.com/paper/2502.02723/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2025.

## Built on by (most cited first, 30 of 48)

- [KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems](https://mlsanity.com/paper/2510.12872/) (2025)
- [Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play](https://mlsanity.com/paper/2509.25541/) (2025)
- [QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models](https://mlsanity.com/paper/2510.16292/) (2025)
- [SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression](https://mlsanity.com/paper/2602.03051/) (2026)
- [LittleBit: Ultra Low-Bit Quantization via Latent Factorization](https://mlsanity.com/paper/2506.13771/) (2025)
- [FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models](https://mlsanity.com/paper/2508.01506/) (2025)
- [A3 : an Analytical Low-Rank Approximation Framework for Attention](https://mlsanity.com/paper/2505.12942/) (2025)
- [Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models](https://mlsanity.com/paper/2505.17974/) (2025)
- [Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression](https://mlsanity.com/paper/2602.02848/) (2026)
- [AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression](https://mlsanity.com/paper/2604.02119/) (2026)
- [CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning](https://mlsanity.com/paper/2509.22075/) (2025)
- [UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs](https://mlsanity.com/paper/2512.03383/) (2025)
- [PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpoint](https://mlsanity.com/paper/2509.10971/) (2025)
- [CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention](https://mlsanity.com/paper/2603.17946/) (2026)
- [NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training](https://mlsanity.com/paper/2603.03597/) (2026)
- [Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression](https://mlsanity.com/paper/2606.03465/) (2026)
- [Understanding Post-Training Structural Changes in Large Language Models](https://mlsanity.com/paper/2509.17866/) (2025)
- [Low-Rank Prehab: Preparing Neural Networks for SVD Compression](https://mlsanity.com/paper/2512.01980/) (2025)
- [Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixing](https://mlsanity.com/paper/2512.03062/) (2025)
- [WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models](https://mlsanity.com/paper/2604.02570/) (2026)
- [Timber: Training-free Instruct Model Refining with Base via Effective Rank](https://mlsanity.com/paper/2509.23595/) (2025)
- [BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression](https://mlsanity.com/paper/2509.25136/) (2025)
- [ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression](https://mlsanity.com/paper/2602.11008/) (2026)
- [Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression](https://mlsanity.com/paper/2604.01609/) (2026)
- [EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture](https://mlsanity.com/paper/2605.24144/) (2026)
- [UniSVQ: 2-bit Unified Scalar-Vector Quantization](https://mlsanity.com/paper/2606.10520/) (2026)
- [MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation](https://mlsanity.com/paper/2506.01897/) (2025)
- [Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reports](https://mlsanity.com/paper/2508.15845/) (2025)
- [Neuronal Group Communication for Efficient Neural representation](https://mlsanity.com/paper/2510.16851/) (2025)
- [COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression](https://mlsanity.com/paper/2602.15200/) (2026)

## Abstract (arXiv)

We provide a new LLM-compression solution via SVD, unlocking new possibilities for LLM compression beyond quantization and pruning. We point out that the optimal use of SVD lies in truncating activations, rather than merely using activations as an optimization distance. Building on this principle, we address three critical challenges in SVD-based LLM compression: including (1) How can we determine the optimal activation truncation position for each weight matrix in LLMs? (2) How can we efficiently reconstruct the weight matrices based on truncated activations? (3) How can we address the inherent "injection" nature that results in the information loss of the SVD? We propose Dobi-SVD, which establishes a new, principled approach to SVD-based LLM compression.