Data to 7 Oct 2026

ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models

arXiv 2312.05821 ↗First posted 10 Dec 2023cs.CLcited by 156

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

In this paper, we introduce a new post-training compression paradigm for Large Language Models (LLMs) to facilitate their wider adoption. We delve into LLM weight low-rank decomposition, and find that the challenges of this task stem from (1) the distribution variance in the LLM activations and (2) the sensitivity difference among various kinds of layers. To address these issues, we propose a training-free approach called Activation-aware Singular Value Decomposition (ASVD). Specifically, ASVD manages activation outliers by transforming the weight matrix based on the activation distribution. This transformation allows the outliers in the activation matrix to be absorbed into the transformed weight matrix, thereby enhancing decomposition accuracy. Additionally, we propose an efficient iterative calibration process to optimize layer-specific decomposition by addressing the varying sensitivity of different LLM layers. In this way, ASVD can compress a network by 10%-30%. Based on the success of the low-rank decomposition of projection matrices in the self-attention module, we further introduce ASVD to compress the KV cache. By reducing the channel dimension of KV activations, memory requirements for KV cache can be largely reduced. ASVD can further achieve 50% KV cache reductions without performance drop in a training-free manner.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 53 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

35 of 53 references.

References · 35

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2020Measuring Massive Multitask Language Understandingcited by 7,768
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2016Pointer Sentinel Mixture Modelscited by 2,992
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234

23 more

2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
2020Efficient Transformers: A Surveycited by 784
2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
2023A Survey on Model Compression for Large Language Modelscited by 263
2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
2022Language model compression with weighted low-rank factorizationcited by 192
2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
2023PB-LLM: Partially Binarized Large Language Modelscited by 80
2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
2021Initialization and Regularization of Factorized Neural Layerscited by 58
2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
2015Training CNNs with Low-Rank Filters for Efficient Image Classificationcited by 25
2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
2017Coordinating Filters for Faster Deep Neural Networkscited by 24
2015ACDC: A Structured Efficient Linear Layercited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 64; 2026: 59.

156 papers.

Sort:

By area:

2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
2024Matryoshka Multimodal Modelscited by 82
2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 77
2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46

38 more

2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantizationcited by 35
2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
2025Delta Decompression for MoE-based LLMs Compressioncited by 32
2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 30
2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 19
2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 16
2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
2024Beware of Calibration Data for Pruning Large Language Modelscited by 14
2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
2025Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decompositioncited by 12
2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
2025Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Modelscited by 11
2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 10
2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 9
2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 9
2024KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Modelscited by 8
2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7
2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensatorscited by 7
2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 7
2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 7
2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
2025SALS: Sparse Attention in Latent Space for KV cache Compressioncited by 7

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way