# Beyond Uniform SVD:Dual-Level Optimization across Columns and Modules for LLM Compression

arXiv: https://arxiv.org/abs/2510.19385 · HTML: https://mlsanity.com/paper/2510.19385/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024)
- [SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression](https://mlsanity.com/paper/2503.12340/) (2025)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation](https://mlsanity.com/paper/2402.16880/) (2024)
- [Revisiting Knowledge Distillation for Autoregressive Language Models](https://mlsanity.com/paper/2402.11890/) (2024)
- [BlockPruner: Fine-grained Pruning for Large Language Models](https://mlsanity.com/paper/2406.10594/) (2024)
- [Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression](https://mlsanity.com/paper/2410.03765/) (2024)
- [Numerical Optimizations for Weighted Low-rank Estimation on Language Model](https://mlsanity.com/paper/2211.09718/) (2022)

## Abstract (arXiv)

Low-rank decomposition, particularly Singular Value Decomposition (SVD), is a pivotal technique for mitigating the storage and computational demands of Large Language Models (LLMs). However, prevalent SVD-based approaches overlook the critical phenomenon that decomposition errors exhibit significant disparity across different components of the parameter matrix, often leading to suboptimal approximation. Furthermore, existing methods lack a direct metric to evaluate the importance of individual weight matrices. To address these limitations, we propose Duo-SVD (Dual-level Optimization SVD), a novel training-free framework that synergizes optimization at both the column and the module levels. First, Duo-SVD incorporates a Column-Preserving Strategy that explicitly retains columns exhibiting high decomposition errors, while applying low-rank approximation solely to those with lower errors. Second, at the module level, we employ a Module-Adaptive Allocation Strategy that formulates ratio allocation as a global constrained optimization problem based on perturbation-induced model deviation. Extensive experiments demonstrate that Duo-SVD consistently outperforms state-of-the-art SVD-based baselines and structured pruning methods, establishing it as a superior paradigm for efficient LLM compression.