mlsanityData to 7 Oct 2026

Rethinking LoRA Memory Through the Lens of KV Cache Compression

arXiv 2606.05698 ↗First posted 4 Jun 2026cs.CLcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Parametric retrieval augmentation encodes document information into lightweight, document-specific modules such as LoRA adapters, reducing the need to include all evidence as input context. However, it remains unclear how this parameter-side memory interacts with context-side memory stored in the KV cache. We study this interaction in document-level question answering by progressively evicting document key-value states and measuring when a document LoRA contributes beyond the retained context. We find that document LoRA adds little when the KV cache is largely intact, but becomes increasingly useful under aggressive compression, recovering 13-21 ROUGE-L points when no document context remains. The gain is largest when the base model encodes the document, and the adapter is applied only during answer generation, suggesting that document LoRA is better understood as decoding-time parametric memory than as a document encoder. Finally, QA-style supervision produces substantially stronger adapters than raw-context next-token-prediction. These results position document LoRA as a complementary memory channel whose value emerges precisely when context-side evidence is scarce.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 29 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

19 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  2. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores10 Jul
  3. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  4. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 12

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  6. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  8. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  9. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancementcited by 24
  10. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 11
  11. 2025LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Taskscited by 7
  12. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRAcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Self-Compacting Language Model Agentscited by 11
  2. 2026Memory Depth, Not Memory Access: Selective Parametric Consolidation for Long-Running Language Agentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
  2. 2024Quantifying reliance on external information over parametric knowledge during Retrieval Augmented Generation (RAG) using mechanistic analysis
  3. 2026Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models
  4. 2026SemPIC: Learning Semantic Position-Independent KV Caches
  5. 2026Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
  6. 2026Cartridges++: KV Cache Compression without Off-Context Derailment
  7. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
  8. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models