mlsanityData to 14 Oct 2026

KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

arXiv 2607.24260 ↗First posted 27 Jul 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 41 references that are in the map. First posted on arXiv 27 Jul 2026.

Builds on

37 of 41 references.

References · 37

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  10. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
25 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  2. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2022Efficiently Scaling Transformer Inferencecited by 539
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  9. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  13. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  14. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  15. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  16. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  17. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 162
  18. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  19. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  20. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 63
  21. 2019Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphscited by 53
  22. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 27
  23. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
  24. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 19
  25. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
  2. 2026ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning
  3. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference
  4. 2026RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference
  5. 2026Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning
  6. 2026Akashic: A Low-Overhead LLM Inference Service with MemAttention
  7. 2026TokenMem: Faithful Knowledge Injection for Frozen LLMs
  8. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs