The Optimization Landscape of Learning Compacted Context Models
On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference
Abstract (arXiv)
Many works approach continual learning through the lens of infinite context windows. As an agent puts more observation into context (concretely the KV cache), compacting said context is akin to direct memory manipulation, without affecting the base model's weights. Many works pose KV compaction as an optimization problem: learn a smaller set of KV vectors that matches the behavior of the full KV cache. While this preserves base model behavior, optimizing through a frozen base model results in a highly nontrivial optimization problem with a brittle and flat loss landscape. In this paper, we characterize what makes these optimization problems difficult and demonstrate that a heavily simplified Perceiver-based architecture not only matches performance of a full Perceiver transformer in continuous context compaction, but outperforms baselines on compaction utility. Results are presented on MCQ tasks across Finance, Legal, Gutenberg, and Code.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 26 references that are in the map. First posted on arXiv 5 Oct 2026.
Builds on
23 of 26 references.
References · 23
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
- 2023Learning to Compress Prompts with Gist Tokenscited by 390
- 2023Adapting Language Models to Compress Contextscited by 356
- 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
- 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
- 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
- 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
11 more
- 2026Fast KV Compaction via Attention Matchingcited by 31
- 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 19
- 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19
- 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 17
- 2023PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizercited by 15
- 2026Learning to Evict from Key-Value Cachecited by 14
- 2026δ-mem: Efficient Online Memory for Large Language Modelscited by 13
- 2026Kwai Summary Attention Technical Reportcited by 7
- 2026Still: Amortized KV Cache Compaction in a Single Forward Passcited by 4
- 2026End-to-End Context Compression at Scalecited by 4
- 2026KVSculpt: KV Cache Compression as Distillationcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
- 2026Persistent Context Graphs for Efficient Memory Compaction in LLM Agents
- 2026LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via State Proprioception
- 2026KV-streams for Efficient Compaction in Agentic Reinforcement Learning
- 2026Beyond Compaction: Structured Context Eviction for Long-Horizon Agents
- 2026PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning
- 2026Continuous Context Management
- 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents