Data to 14 Oct 2026

SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression

arXiv 2503.12340 ↗First posted 16 Mar 2025cs.CLcited by 67

Not on the map. Why?

Abstract (arXiv)

Despite significant advancements, the practical deployment of Large Language Models (LLMs) is often hampered by their immense sizes, highlighting the need for effective compression techniques. Singular Value Decomposition (SVD) is a promising LLM compression technique. However, existing SVD-based compression methods fall short in reducing truncation losses, leading to less competitive performance in compressed models. In this work, we introduce SVD-LLM V2, a SVD-based LLM compression method that optimizes singular value truncation in SVD compression with two techniques. First, SVD-LLM V2 proposes to use theoretical truncation loss of weight matrices to assign a unique compression ratio to each weight matrix at different layers to accommodate weight redundancy heterogeneity. Second, SVD-LLM V2 proposes loss-optimized weight truncation to ensure that the truncated singular values result in a lower and more stable truncation loss in practice. We evaluate SVD-LLM V2 on ten datasets and five LLMs at various scales. Our results show SVD-LLM V2 outperforms state-of-the-art SVD-based LLM compression methods. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

27 of 34 references.

References · 27

2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
2016Pointer Sentinel Mixture Modelscited by 3,096
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
2023A Survey of Large Language Modelscited by 2,576
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919

15 more

2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
2022Language model compression with weighted low-rank factorizationcited by 195
2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
2024What Matters in Transformers? Not All Attention is Neededcited by 84
2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
2025Efficient Diffusion Models: A Surveycited by 35
2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
2023An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applicationscited by 20
2024The Internet of Things in the Era of Generative AI: Vision and Challengescited by 8
2024Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retrainingcited by 8
2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusioncited by 8
2024ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantizationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 45.

67 papers.

Sort:

By area:

2025AI Flow: Perspectives, Scenarios, and Approachescited by 44
2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 16
2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 12
2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
2025FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Modelscited by 9
2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 8
2026Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compressioncited by 8
2026Model Merging in the Essential Subspacecited by 7
2026AA-SVD : Anchored and Adaptive SVD for Large Language Model Compressioncited by 7
2025CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learningcited by 6

38 more

2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 5
2025COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximationcited by 5
2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attentioncited by 5
2025Addition is almost all you need: Compressing large language models with double binary factorizationcited by 3
2025Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLMcited by 3
2025Towards Reversible Model Merging For Low-rank Weightscited by 3
2026WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Modelscited by 3
2026FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fastcited by 3
2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
2026FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deploymentcited by 2
2026Ruyi2 Technical Reportcited by 2
2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compressioncited by 2
2026UniSVQ: 2-bit Unified Scalar-Vector Quantizationcited by 2
2025S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Plannercited by 1
2025Neural Weight Compression for Language Modelscited by 1
2025Neuronal Group Communication for Efficient Neural representationcited by 1
2025WARP: Weight Teleportation for Attack-Resilient Unlearning Protocolscited by 1
2025SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skippingcited by 1
2026COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compressioncited by 1
2026Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compressioncited by 1
2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
2026Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregationcited by 1
2026SLORR: Simple and Efficient In-Training Low-Rank Regularizationcited by 1
2025Beyond Uniform SVD:Dual-Level Optimization across Columns and Modules for LLM Compressioncited by 0
2025NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainiumcited by 0
2025Optimal Singular Damage: Efficient LLM Inference in Low Storage Regimescited by 0
2026Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deploymentcited by 0
2026MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Modelscited by 0
2026SPQ: An Ensemble Technique for Large Language Model Compressioncited by 0
2026SVD Contextual Sparsity Predictors for Fast LLM Inferencecited by 0
2026Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcyclingcited by 0
2026DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Modelscited by 0
2026Predicting LLM Compression Degradation from Spectral Statisticscited by 0
2026IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compressioncited by 0
2026SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Modelscited by 0
2026Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limitscited by 0

50 most cited of 67 All 67 (JSON)