mlsanityData to 7 Oct 2026

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

arXiv 2609.38169 ↗First posted 29 Sep 2026cs.CL cs.AI cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 54 of its 67 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

54 of 67 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization27 Aug
  2. 2026DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization20 Apr
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 49

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
37 more
  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  7. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  8. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  10. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  11. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  12. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  13. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  14. 2025Generalizing Verifiable Instruction Followingcited by 178
  15. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  16. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  17. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  18. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  19. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 115
  20. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  21. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  22. 2021OMPQ: Orthogonal Mixed Precision Quantizationcited by 37
  23. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  24. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  25. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  26. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 25
  27. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  28. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  29. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantizationcited by 19
  30. 2023Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspectivecited by 17
  31. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
  32. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  33. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 10
  34. 2026Quantized Reasoning Models Think They Need to Think Longer, but They Do Notcited by 10
  35. 2026AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafterscited by 4
  36. 2026Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Modelscited by 3
  37. 2026Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressedcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
  2. 2026Runtime-Certified Bounded-Error Quantized Attention
  3. 2026Low-Discrepancy Dither for Quantized Recurrent State Caches
  4. 2026Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
  5. 2026QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
  6. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  7. 2026KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
  8. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs