mlsanityData to 14 Oct 2026

Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

arXiv 2609.04748 ↗First posted 4 Sep 2026cs.SE cs.DC cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Major open-source serving stacks reuse the key-value tensors of a shared prefix by default. We measure what this prefix reuse costs in reproducibility. Holding model, decoding, seed and request order fixed at batch size one, we ran an eighty-episode multi-turn agentic tool-use workload with prefix reuse on and off across two engines and four weight formats. Without prefix reuse, repeated runs reproduced every token and log-probability in all ten configurations. With it, cache-hit and recompute paths diverged on both engines. On llama.cpp with GGUF weights this path divergence rises with quantization: for Qwen2.5-7B over ten independent server histories, it reached 37.4 percent of episodes at 16-bit and 82.5 percent at four-bit. Near-tie decisions became only about 1.2 times as common, while the cache-path change in top-two margins grew 4.9-fold from 16-bit to three-bit. Serving four-bit weights through 16-bit kernels left divergence not detectably different from 16-bit, placing the amplification mainly in the quantized execution path. At Q4_K_M, run-to-run divergence came mainly from llama.cpp's server-level prompt cache: under the canonical episode order it moved divergence by 37.5 points, and at its default, divergence spanned 20.0 to 28.7 percent under five random orders. With cache state restored, each path reproduced exactly while the two still differed. In the tested stacks, cached serving is irreproducible because the cache state is absent from the request and not reset by default. A single-turn bridge shows path divergence reaching task outcomes with no detectable net change in accuracy. We release the harness, logs, and analysis pipeline.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 36 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

24 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference16 Apr
  2. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  6. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  7. 2024AI Agents That Mattercited by 219
  8. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  9. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  10. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  11. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  12. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflowscited by 57
8 more
  1. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 53
  2. 2024How Does Quantization Affect Multilingual LLMs?cited by 43
  3. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  4. 2024Accuracy is Not All You Needcited by 31
  5. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 19
  6. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 16
  7. 2026The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibilitycited by 8
  8. 2026Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
  2. 2026Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
  3. 2026When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
  4. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
  5. 2026Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches
  6. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  7. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  8. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents