mlsanityData to 14 Oct 2026

Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

arXiv 2502.02723 ↗First posted 4 Feb 2025cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

We provide a new LLM-compression solution via SVD, unlocking new possibilities for LLM compression beyond quantization and pruning. We point out that the optimal use of SVD lies in truncating activations, rather than merely using activations as an optimization distance. Building on this principle, we address three critical challenges in SVD-based LLM compression: including (1) How can we determine the optimal activation truncation position for each weight matrix in LLMs? (2) How can we efficiently reconstruct the weight matrices based on truncated activations? (3) How can we address the inherent "injection" nature that results in the information loss of the SVD? We propose Dobi-SVD, which establishes a new, principled approach to SVD-based LLM compression.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 28.

48 papers.

Sort:

By area:

  1. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 40
  2. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  3. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
  4. 2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 12
  5. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  6. 2025FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Modelscited by 9
  7. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 8
  8. 2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 8
  9. 2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 8
  10. 2026AA-SVD : Anchored and Adaptive SVD for Large Language Model Compressioncited by 7
  11. 2025CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learningcited by 6
  12. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
36 more
  1. 2025PHLoRA: data-free Post-hoc Low-Rank Adapter extraction from full-rank checkpointcited by 5
  2. 2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attentioncited by 5
  3. 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Trainingcited by 4
  4. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compressioncited by 4
  5. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 3
  6. 2025Low-Rank Prehab: Preparing Neural Networks for SVD Compressioncited by 3
  7. 2025Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixingcited by 3
  8. 2026WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Modelscited by 3
  9. 2025Timber: Training-free Instruct Model Refining with Base via Effective Rankcited by 2
  10. 2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
  11. 2026ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compressioncited by 2
  12. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compressioncited by 2
  13. 2026EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecturecited by 2
  14. 2026UniSVQ: 2-bit Unified Scalar-Vector Quantizationcited by 2
  15. 2025MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptationcited by 1
  16. 2025Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reportscited by 1
  17. 2025Neuronal Group Communication for Efficient Neural representationcited by 1
  18. 2026COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compressioncited by 1
  19. 2026Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compressioncited by 1
  20. 2026Dynamic Model Merging Made Slimcited by 1
  21. 2026SLORR: Simple and Efficient In-Training Low-Rank Regularizationcited by 1
  22. 2025IPPRO: Importance-based Pruning with PRojective Offset for Magnitude-indifferent Structural Pruningcited by 0
  23. 2025NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainiumcited by 0
  24. 2026IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compressioncited by 0
  25. 2026SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Modelscited by 0
  26. 2026LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Modelscited by 0
  27. 2026Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomiescited by 0
  28. 2026UniRank: Unified Rank Allocation for Low-Rank LLM Compressioncited by 0
  29. 2026LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compressioncited by 0
  30. 2026Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clientscited by 0
  31. 2026LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compressioncited by 0
  32. 2026Importance-Aware Low-Rank Distillation of Diffusion Transformerscited by 0
  33. 2026Mind the Approximation: Fisher-Weighted SVD Compression for ViTscited by 0
  34. 2026The Functional Structure of Post-Compression Recovery in Low-Rank LLMscited by 0
  35. 2026Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weightscited by 0
  36. 2026GA-SVD: Generation-aware Singular Value Decomposition for LLM Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Differentiable Bit-Widths: Co-optimizing Pruning and Quantization via SVD for Ultra-Efficient LLM Compression
  2. 2025DipSVD: Dual-importance Protected SVD for Efficient LLM Compression
  3. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
  4. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
  5. 2026HAS-VQ: Hessian-Adaptive Sparse Vector Quantization for High-Fidelity LLM Compression
  6. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
  7. 2025Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
  8. 2025AdaSVD: Adaptive Singular Value Decomposition for Large Language Models