Data to 7 Oct 2026

SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

arXiv 2403.07378 ↗First posted 12 Mar 2024cs.CL cs.LGcited by 231

Not on the map. Why?

Cited by 30 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitates LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening technique to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a parameter update with sequential low-rank approximation to compensate for the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10 datasets and seven models from three different LLM families at three different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

37 of 57 references.

References · 37

2020Language Models are Few-Shot Learnerscited by 32,199
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
2016Pointer Sentinel Mixture Modelscited by 2,992
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818

25 more

2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
2023A Survey of Large Language Modelscited by 2,537
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
2015A large annotated corpus for learning natural language inferencecited by 1,837
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
2018Neural Network Acceptability Judgmentscited by 952
2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
2023A Survey on Model Compression for Large Language Modelscited by 263
2022Language model compression with weighted low-rank factorizationcited by 192
2023Efficient Large Language Models: A Surveycited by 183
2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 110
2023PB-LLM: Partially Binarized Large Language Modelscited by 80
2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applicationscited by 20
2024The Internet of Things in the Era of Generative AI: Vision and Challengescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 99; 2026: 106.

231 papers.

Sort:

By area:

2024A Survey on Efficient Inference for Large Language Modelscited by 197
2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 63
2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 39
2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31

38 more

2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 26
2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
2025Principled Multimodal Representation Learningcited by 19
2025LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptationcited by 17
2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 16
2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 14
2024Beware of Calibration Data for Pruning Large Language Modelscited by 14
2025Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledgecited by 14
2024SLIP: Securing LLMs IP Using Weights Decompositioncited by 13
2025Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decompositioncited by 12
2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
2025Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoningcited by 11
2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 10
2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 9
2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusioncited by 8
2024ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantizationcited by 8
2024KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Modelscited by 8
2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
2025FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Modelscited by 8
2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusioncited by 8
2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 7
2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 7
2025DipSVD: Dual-importance Protected SVD for Efficient LLM Compressioncited by 7
2025LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkitcited by 7
2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 7
2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 7
2024EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximationcited by 6
2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 6

50 most cited of 231 All 231 (JSON)