PReM: Learning What to Preserve and When to Refresh for Context Compression
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Efficient long-context inference is not only about reducing memory cost, but also about keeping useful contextual evidence accessible as generation proceeds. However, existing compression-oriented approaches, such as key-value (KV) cache compression and context compression, often either make an early decision about which contextual information to keep or rely on an external compressor. Such designs make it difficult to adapt the compressed context to the evidence needed by later reasoning steps. This paper introduces PReM (Preserve and Refresh Memory), a context-compression framework that maintains the long context as the model's internal layer-wise KV memory and learns what to preserve and when to refresh it. Specifically, PReM uses a dedicated memory layer to make memory-selection decisions, and a special memory token <m> to trigger refreshes during generation. To train this behavior, PReM introduces Phase-Separated Refresh Training, aligning memory selection with memory-conditioned generation while preserving continuity across refreshes. Experiments with 32K-token contexts show that PReM outperforms strong baselines under both 16x and 32x compression, while maintaining a favorable balance between answer quality and inference efficiency.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 27 references that are in the map. First posted on arXiv 15 Jul 2026.
Builds on
16 of 27 references.
Closest ancestors · 3
Same direction, newest first
- 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
Other references · 13
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
- 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
- 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
- 2024Long Context Compression with Activation Beaconcited by 104
- 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
- 2026IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inferencecited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024RefreshKV: Updating Small KV Cache During Long-form Generation
- 2024Characterizing Prompt Compression Methods for Long Context Inference
- 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memory
- 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction
- 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
- 2026Behavior-Preserving KV Cache Compression
- 2026HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
- 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference