# Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs

arXiv: https://arxiv.org/abs/2608.08506 · HTML: https://mlsanity.com/paper/2608.08506/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models](https://mlsanity.com/paper/2312.05821/) (2023)
- [GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration](https://mlsanity.com/paper/2504.02692/) (2025)
- [On the Impact of Calibration Data in Post-training Quantization and Pruning](https://mlsanity.com/paper/2311.09755/) (2023)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization](https://mlsanity.com/paper/2504.09629/) (2025)
- [The LLM Surgeon](https://mlsanity.com/paper/2312.17244/) (2023)
- [Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression](https://mlsanity.com/paper/2410.03765/) (2024)
- [Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization](https://mlsanity.com/paper/2505.11695/) (2025)
- [SlimLLM: Accurate Structured Pruning for Large Language Models](https://mlsanity.com/paper/2505.22689/) (2025)
- [You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning](https://mlsanity.com/paper/2501.15296/) (2025)
- [ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference](https://mlsanity.com/paper/2511.10645/) (2025)
- [QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models](https://mlsanity.com/paper/2510.16292/) (2025)
- [Beware of Calibration Data for Pruning Large Language Models](https://mlsanity.com/paper/2410.17711/) (2024)
- [AdaSVD: Adaptive Singular Value Decomposition for Large Language Models](https://mlsanity.com/paper/2502.01403/) (2025)

## Built on by (most cited first, 2 of 2)

- [LaMoC: Loss-Aware Modular Compression for LLMs](https://mlsanity.com/paper/2608.30226/) (2026)
- [Learning Functional Subspaces for Neural Network Compression](https://mlsanity.com/paper/2609.40127/) (2026)

## Abstract (arXiv)

Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy. However, existing SOTA frameworks share two key limitations: (1) residual errors in calibration data activations accumulate across layers during compression, causing misalignment between representations simulated at compression time and those experienced at inference; (2) the assumption that layer importance distribution is preserved post-compression does not hold. Together, these two effects introduce misalignment in the compression process in relation to the deployed model. We study these effects and propose a simple, training-free methodology compatible with existing frameworks to mitigate them, comprising: (1) Layer-by-Layer Compression with Calibration Correction; (2) Iterative Compression with Rank Allocation Correction. Implemented atop an existing SOTA decomposition framework, and evaluated on Llama and Qwen3 models across various benchmarks and compression rates, our approach demonstrates up to ~1-2.5 accuracy point improvements over per-weight and joint decomposition baselines on zero-shot tasks.