# Latent Context Compilation: Distilling Long Context into Compact Portable Memory

arXiv: https://arxiv.org/abs/2602.21221 · HTML: https://mlsanity.com/paper/2602.21221/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [Test-Time Training on Nearest Neighbors for Large Language Models](https://mlsanity.com/paper/2305.18466/) (2023)
- [Test-Time Model Adaptation with Only Forward Passes](https://mlsanity.com/paper/2404.01650/) (2024)
- [Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution](https://mlsanity.com/paper/2510.00636/) (2025)
- [ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2502.00299/) (2025)
- [Test-Time Learning for Large Language Models](https://mlsanity.com/paper/2505.20633/) (2025)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)
- [Compressed Context Memory For Online Language Model Interaction](https://mlsanity.com/paper/2312.03414/) (2023)
- [With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation](https://mlsanity.com/paper/2401.11504/) (2024)
- [A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression](https://mlsanity.com/paper/2412.17483/) (2024)
- [InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation](https://mlsanity.com/paper/2504.01707/) (2025)
- [Context Cascade Compression: Exploring the Upper Limits of Text Compression](https://mlsanity.com/paper/2511.15244/) (2025)
- [StreamAdapter: Efficient Test Time Adaptation from Contextual Streams](https://mlsanity.com/paper/2411.09289/) (2024)

## Built on by (most cited first, 13 of 13)

- [Context Distillation as Latent Memory Management](https://mlsanity.com/paper/2605.28889/) (2026)
- [RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction](https://mlsanity.com/paper/2608.01247/) (2026)
- [A Survey on Self-Improving Test-Time Intelligence: Feedback-Driven Adapting, Learning, and Scaling at Inference](https://mlsanity.com/paper/2609.01679/) (2026)
- [Context Compaction Theory](https://mlsanity.com/paper/2608.01326/) (2026)
- [Cartridges++: KV Cache Compression without Off-Context Derailment](https://mlsanity.com/paper/2609.35621/) (2026)
- [Large Language Models Do Not Always Need Readable Language](https://mlsanity.com/paper/2606.19857/) (2026)
- [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://mlsanity.com/paper/2607.02512/) (2026)
- [One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA](https://mlsanity.com/paper/2606.10572/) (2026)
- [Compact-Memory LLM Agents via Online Max-Member Clustering and Atom-Aware Packing](https://mlsanity.com/paper/2609.04915/) (2026)
- [Machine-Interpretable Information: Compiling Documents into Searchable and Readable Protocol States](https://mlsanity.com/paper/2609.23371/) (2026)
- [MemFold: Learning Compact Soft Memory for Long-Context Personalization via On-Policy Optimization](https://mlsanity.com/paper/2609.36435/) (2026)
- [When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attention](https://mlsanity.com/paper/2609.36766/) (2026)
- [Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning](https://mlsanity.com/paper/2610.11784/) (2026)

## Abstract (arXiv)

Efficient long-context LLM deployment is stalled by a dichotomy between amortized compression, which struggles with out-of-distribution generalization, and Test-Time Training, which incurs prohibitive synthetic data costs and requires modifying model weights, creating stateful parameters that complicate concurrent serving. We propose Latent Context Compilation, a framework that fundamentally shifts context processing from adaptation to compilation. By utilizing a disposable LoRA module as a compiler, we distill long contexts into compact buffer tokens -- stateless, portable memory artifacts that are plug-and-play compatible with frozen base models. Crucially, we introduce a self-aligned optimization strategy that eliminates the need for synthetic context-relevant QA pairs. By regularizing context reconstruction task with context-agnostic random queries, we force compressed tokens to reside within the model's existing instruction-following manifold. Experiments with Llama-3.1-8B demonstrate that Latent Context Compilation preserves fine-grained details and reasoning capabilities where prior methods falter, effectively decoupling memory density from model parameters even at a 16x compression ratio.