Knowledge Packs: Zero-Token Knowledge Delivery via KV Cache Injection
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
RAG wastes tokens. We propose Knowledge Packs: pre-computed KV caches that deliver the same knowledge at zero token cost. For causal transformers, the KV cache from a forward pass on text F is identical to what a joint pass on F+q would produce - this follows directly from the causal mask. The equivalence is exact but fragile: wrong chat template formatting causes 6-7pp degradation, which we believe explains prior claims of KV outperforming RAG. With correct formatting: zero divergences across 700 questions on Qwen3-8B and Llama-3.1-8B, up to 95% token savings. The KV interface also enables behavioral steering that RAG cannot do. Because RoPE rotates keys but leaves values untouched, contrastive deltas on cached values can nudge model behavior while key arithmetic destroys coherence. The effect sits in mid-layer values (33-66%), independent directions are nearly orthogonal (cos~0) and compose, and both channels - knowledge and steering - run simultaneously at alpha<=0.7 without interference. No training, no weight modification.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 21 references that are in the map. First posted on arXiv 22 Mar 2026.
Builds on
18 of 21 references.
Closest ancestors · 4
Same direction, newest first
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May
Other references · 14
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
- 2022Mass-Editing Memory in a Transformercited by 976
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
- 2026Probing the Prompt KV Cache: Where It Becomes Dispensable
- 2026Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoning
- 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs
- 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
- 2026Minimal-Intervention KV Retention via Set-Conditioned Diversity
- 2026Mask-Guided KV Cache Eviction in Block Diffusion Language Models
- 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs