# Preserve-Then-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMs

arXiv: https://arxiv.org/abs/2602.02001 · HTML: https://mlsanity.com/paper/2602.02001/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 51 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 30 of 36)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models](https://mlsanity.com/paper/2310.08659/) (2023)
- [RedPajama: an Open Dataset for Training Large Language Models](https://mlsanity.com/paper/2411.12372/) (2024)
- [ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models](https://mlsanity.com/paper/2312.05821/) (2023)
- [Fixed Point Quantization of Deep Convolutional Networks](https://mlsanity.com/paper/1511.06393/) (2015)
- [RPTQ: Reorder-based Post-training Quantization for Large Language Models](https://mlsanity.com/paper/2304.01089/) (2023)
- [LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning](https://mlsanity.com/paper/2311.12023/) (2023)
- [Compressing Large Language Models using Low Rank and Low Precision Decomposition](https://mlsanity.com/paper/2405.18886/) (2024)

## Built on by (most cited first, 1 of 1)

- [GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation](https://mlsanity.com/paper/2606.01412/) (2026)

## Abstract (arXiv)

Quantization Error Reconstruction (QER) reduces accuracy loss in Post-Training Quantization (PTQ) by approximating weights as W ≈ Q + LR, using a rank-r correction to reconstruct quantization error. Prior methods devote the full rank budget to error reconstruction, which is suboptimal when W has intrinsic low-rank structure and quantization corrupts dominant directions. We propose Structured Residual Reconstruction (SRR), a rank-allocation framework that preserves the top-k singular subspace of the activation-scaled weight before quantization, quantizes only the residual, and uses the remaining rank r-k for error reconstruction. We derive a theory-guided criterion for selecting k by balancing quantization-exposed energy and unrecoverable error under rank constraints. We further show that resulting Q + LR parameterization naturally supports Quantized Parameter-Efficient Fine-Tuning (QPEFT), and stabilizes fine-tuning via gradient scaling along preserved directions. Experiments demonstrate consistent perplexity reductions across diverse models and quantization settings in PTQ, along with a 5.9 percentage-point average gain on GLUE under 2-bit QPEFT. The project page is available at https://ai-isl.github.io/srr.