mlsanityData to 7 Oct 2026

Compression-Aware Abstention: Teaching LLMs to Refuse When KV-Compression Masks Remove Answer Evidence

arXiv 2608.29934 ↗First posted 30 Aug 2026cs.CL cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

KV-cache compression reduces LLM inference memory by evicting context tokens, but when the evicted tokens contain answer-bearing evidence, the model may hallucinate instead of recognizing that the compressed context is insufficient. We address this failure from a behavioral perspective: to our knowledge, this is the first work to formulate compression-aware abstention as a learning problem, in which a model learns to answer when supporting evidence survives compression and abstain when it does not. We construct supervision from compressor survival masks and tight answer-bearing spans, labeling examples as Confident when evidence survives and Abstain when it is removed. A 10.1M-parameter LoRA adapter trained on ~2.6K MuSiQue 2-hop QA examples reduces base-model hallucinations by 97% under prompt-style truncation while preserving correct answering on evidence-retaining examples. Unlike prompt-only abstention baselines, which over-abstain on many answerable high-retention examples, the trained adapter learns a conditional policy. We also evaluate the method under actual compressed-cache decoding, where multi-compressor training yields a 6-22x relative lift over the unaided base on evidence-retaining examples. Controlled-deletion experiments show that the learned behavior is driven by evidence content rather than input length alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 29 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

20 of 29 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  2. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2020Selective Question Answering under Domain Shiftcited by 228
  10. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  11. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
  2. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
  3. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compression
  4. 2026SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
  5. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
  6. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  7. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
  8. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference