mlsanityData to 14 Oct 2026

Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs

arXiv 2608.08506 ↗First posted 9 Aug 2026cs.AI cs.CL cs.PFcited by 2

Not on the map. Why?

Abstract (arXiv)

Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy. However, existing SOTA frameworks share two key limitations: (1) residual errors in calibration data activations accumulate across layers during compression, causing misalignment between representations simulated at compression time and those experienced at inference; (2) the assumption that layer importance distribution is preserved post-compression does not hold. Together, these two effects introduce misalignment in the compression process in relation to the deployed model. We study these effects and propose a simple, training-free methodology compatible with existing frameworks to mitigate them, comprising: (1) Layer-by-Layer Compression with Calibration Correction; (2) Iterative Compression with Rank Allocation Correction. Implemented atop an existing SOTA decomposition framework, and evaluated on Llama and Qwen3 models across various benchmarks and compression rates, our approach demonstrates up to ~1-2.5 accuracy point improvements over per-weight and joint decomposition baselines on zero-shot tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 9 Aug 2026.

Builds on

40 of 55 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  10. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
28 more
  1. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  2. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  3. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  4. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  5. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  6. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  7. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 58
  8. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  9. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  10. 2023The LLM Surgeoncited by 38
  11. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 35
  12. 2025Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantizationcited by 28
  13. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  14. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 17
  15. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 17
  16. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
  17. 2024Beware of Calibration Data for Pruning Large Language Modelscited by 15
  18. 2025AdaSVD: Adaptive Singular Value Decomposition for Large Language Modelscited by 15
  19. 2025Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probingcited by 14
  20. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  21. 2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 12
  22. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimizationcited by 9
  23. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 7
  24. 2025Optimal Brain Restoration for Joint Quantization and Sparsification of LLMscited by 6
  25. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  26. 2026Cut Less, Fold More: Model Compression through the Lens of Projection Geometrycited by 4
  27. 2026SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compressioncited by 4
  28. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LaMoC: Loss-Aware Modular Compression for LLMscited by 0
  2. 2026Learning Functional Subspaces for Neural Network Compressioncited by 0