mlsanityData to 7 Oct 2026

SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving

arXiv 2606.11244 ↗First posted 4 Jun 2026cs.AR cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Efficient large language model (LLM) serving is increasingly constrained by deployment cost. Quantization is a key technique for reducing serving cost, yet even state-of-the-art 4-bit quantizers exhibit a noticeable quality gap from FP16, particularly for smaller models where low-bit serving is most beneficial. We identify a fundamental cause of this gap: quantization error is highly input-dependent and varies substantially across tokens, while existing post-quantization compensation methods are static and apply identical corrections to all inputs. As a result, easy tokens are over-corrected while hard tokens remain under-corrected. We present SPEAR, a system for post-quantization error-adaptive recovery that improves low-bit LLM serving. SPEAR introduces lightweight Error Compensators (ECs) modulated by per-token gates and places them only at the most error-sensitive layers identified through a CKA-guided entropy-aware diagnostic. This focuses a small parameter budget where it is most effective. Efficient deployment of ECs presents several systems challenges, including additional computation, tensor-parallel synchronization caused by input-dependent gating, and latency instability across configurations. SPEAR addresses these issues through adaptive kernel-fusion dispatch, combining an epilogue-integrated peer-reduction kernel with P2P dual-write to fuse the post-EC computation into low-bit GEMMs, and an SLO-constrained EC-aware scheduler for predictable serving performance. Across challenging per-channel quantization settings, SPEAR recovers 56-75% of the perplexity gap between W4 and FP16 while adding less than 1% model memory overhead and maintaining latency comparable to a widely used 4-bit serving deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 46 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

37 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SpinQuant: LLM quantization with learned rotations26 May
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  4. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
21 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  6. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  8. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  9. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  10. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 54
  11. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 48
  12. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  13. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 39
  14. 2024Mirage: A Multi-Level Superoptimizer for Tensor Programscited by 39
  15. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  16. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  17. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  18. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inferencecited by 22
  19. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 21
  20. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  21. 2024ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantizationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
  2. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?
  3. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
  4. 2023A Speed Odyssey for Deployable Quantization of LLMs
  5. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
  6. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  7. 2025SDQ-LLM: Sigma-Delta Quantization for 1-bit LLMs of any size
  8. 2026Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL