mlsanityData to 14 Oct 2026

Cartridges: Lightweight and general-purpose long context representations via self-study

arXiv 2506.06266 ↗First posted 6 Jun 2025cs.CL cs.AI cs.LGcited by 86

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Large language models are often used to answer queries grounded in large text corpora (e.g. codebases, legal documents, or chat histories) by placing the entire corpus in the context window and leveraging in-context learning (ICL). Although current models support contexts of 100K-1M tokens, this setup is costly to serve because the memory consumption of the KV cache scales with input length. We explore an alternative: training a smaller KV cache offline on each corpus. At inference time, we load this trained KV cache, which we call a Cartridge, and decode a response. Critically, the cost of training a Cartridge can be amortized across all the queries referencing the same corpus. However, we find that the naive approach of training the Cartridge with next-token prediction on the corpus is not competitive with ICL. Instead, we propose self-study, a training recipe in which we generate synthetic conversations about the corpus and train the Cartridge with a context-distillation objective. We find that Cartridges trained with self-study replicate the functionality of ICL, while being significantly cheaper to serve. On challenging long-context benchmarks, Cartridges trained with self-study match ICL performance while using 38.6x less memory and enabling 26.4x higher throughput. Self-study also extends the model's effective context length (e.g. from 128k to 484k tokens on MTOB) and surprisingly, leads to Cartridges that can be composed at inference time without retraining.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 104 of its 121 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

104 of 121 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMs13 Mar
  2. 2024AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models3 Sep
  3. 2024LLoCO: Learning Long Contexts Offline11 Apr
  4. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  5. 2023In-context Autoencoder for Context Compression in a Large Language Model13 Jul

Other references · 99

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2020Longformer: The Long-Document Transformercited by 3,413
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  13. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  14. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  15. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  16. 2016Sequence-Level Knowledge Distillationcited by 801
  17. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  18. 2024Phi-4 Technical Reportcited by 711
  19. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  20. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  21. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  22. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  23. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  24. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  25. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  26. 2023Adapting Language Models to Compress Contextscited by 356
  27. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  28. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  29. 2024Titans: Learning to Memorize at Test Timecited by 328
  30. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  31. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  32. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  33. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  34. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  35. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  36. 2023Using an LLM to Help With Code Understandingcited by 228
  37. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  38. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 71.

86 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Self-Distillation Enables Continual Learningcited by 247
  3. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 173
  4. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  5. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
  6. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  7. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 43
  8. 2026Aligning Language Models from User Interactionscited by 28
  9. 2025MemR3: Memory Retrieval via Reflective Reasoning for LLM Agentscited by 19
  10. 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environmentscited by 17
  11. 2025Pretraining with hierarchical memories: separating long-tail and common knowledgecited by 14
  12. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 13
38 more
  1. 2026Self-Compacting Language Model Agentscited by 11
  2. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  3. 2025Prompt-MII: Meta-Learning Instruction Induction for LLMscited by 9
  4. 2026GradMem: Learning to Write Context into Memory with Test-Time Gradient Descentcited by 9
  5. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  6. 2026Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learningcited by 8
  7. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 8
  8. 2025EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environmentscited by 7
  9. 2026Training Transformers for KV Cache Compressibilitycited by 7
  10. 2026When Does Continual Learning Require Learningcited by 7
  11. 2026Cartridges at Scale: Training Modular KV Caches over Large Document Collectionscited by 6
  12. 2026Do LLMs Benefit From Their Own Words?cited by 5
  13. 2026A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilitiescited by 5
  14. 2026Parallel Context Compaction for Long-Horizon LLM Agent Servingcited by 5
  15. 2026Nearly Optimal Attention Coresetscited by 4
  16. 2026MeMo: Memory as a Modelcited by 4
  17. 2026Still: Amortized KV Cache Compaction in a Single Forward Passcited by 4
  18. 2026End-to-End Context Compression at Scalecited by 4
  19. 2026KV-Skill: Forging Expertise in the Model's Native Languagecited by 4
  20. 2025ARC-Encoder: learning compressed text representations for large language modelscited by 3
  21. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  22. 2026Nectar: Neural Estimation of Cached-Token Attention via Regressioncited by 3
  23. 2026RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Evictioncited by 3
  24. 2025Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representationscited by 2
  25. 2025SnapStream: Efficient Long Sequence Decoding on Dataflow Acceleratorscited by 2
  26. 2026L3: Large Lookup Layerscited by 2
  27. 2026Recursive Language Models Meet Uncertainty: The Surprising Effectiveness of Self-Reflective Program Search for Long Contextcited by 2
  28. 2026SIEVE: Sample-Efficient Parametric Learning from Natural Languagecited by 2
  29. 2026Neural Garbage Collection: Learning to Forget while Learning to Reasoncited by 2
  30. 2026SLIDERS: Systematic Reviews via Automated Evidence Synthesis and Reconciliationcited by 2
  31. 2026Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gapcited by 2
  32. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 2
  33. 2026Doc-to-Atom: Learning to Compile and Compose Memory Atomscited by 2
  34. 2026Practical Online KV Cache Compaction for LLM Agents: An Empirical Studycited by 2
  35. 2026Learning What to Remember: Test-Time Training via Context Distillationcited by 2
  36. 2026Prefix Sliding for efficient test-time scalingcited by 2
  37. 2026Cartridges++: KV Cache Compression without Off-Context Derailmentcited by 2
  38. 2025Mixture of Lookup Key-Value Expertscited by 1

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
  2. 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
  3. 2024ACER: Automatic Language Model Context Extension via Retrieval
  4. 2026Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models
  5. 2026Trained Agentic Context Management
  6. 2022Parallel Context Windows for Large Language Models
  7. 2025Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
  8. 2024QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression