# Learned Structure in Cartridges: Keys as Shareable Routers in Self-Studied Representations

arXiv: https://arxiv.org/abs/2508.17032 · HTML: https://mlsanity.com/paper/2508.17032/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 23 Aug 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [Hydragen: High-Throughput LLM Inference with Shared Prefixes](https://mlsanity.com/paper/2402.05099/) (2024)
- [Learning to Compress Prompt in Natural Language Formats](https://mlsanity.com/paper/2402.18700/) (2024)
- [Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis](https://mlsanity.com/paper/2405.08944/) (2024)
- [When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations](https://mlsanity.com/paper/2310.19698/) (2023)
- [Can Foundation Models Help Us Achieve Perfect Secrecy?](https://mlsanity.com/paper/2205.13722/) (2022)
- [Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs](https://mlsanity.com/paper/2403.08845/) (2024)
- [Verdict: A Library for Scaling Judge-Time Compute](https://mlsanity.com/paper/2502.18018/) (2025)
- [How Relevance Emerges: Interpreting LoRA Fine-Tuning in Reranking LLMs](https://mlsanity.com/paper/2504.08780/) (2025)
- [Mechanistic Interpretability of LoRA-Adapted Language Models for Nuclear Reactor Safety Applications](https://mlsanity.com/paper/2507.09931/) (2025)

## Built on by (most cited first, 2 of 2)

- [Still: Amortized KV Cache Compaction in a Single Forward Pass](https://mlsanity.com/paper/2606.07878/) (2026)
- [Cartridges++: KV Cache Compression without Off-Context Derailment](https://mlsanity.com/paper/2609.35621/) (2026)

## Abstract (arXiv)

A bottleneck for long-context LLM inference is the linearly growing KV cache. Recent work has proposed Cartridges, an approach which leverages offline compute to train a much smaller KV cache than is typically required for a full document (up to 40x less memory usage at inference time). In this paper, we present the first mechanistic exploration of the learned Cartridge key-value cache structure. In particular, we propose that (1) Cartridge keys act as stable, shareable retrieval routers for the compressed corpora and (2) most of the learned compression occurs within the Cartridge value vectors. We present empirical evidence of our routing theory across tasks, model families, and model sizes; for example, we can ablate the learned Cartridge key vectors between tasks with little performance loss. Finally, we propose a slight improvement in initialization called Sampled Chunk Initialization (SCI). We suggest that SCI can lead to faster Cartridge convergence than previously demonstrated in the literature. Our findings lay the groundwork for broader empirical study of Cartridge training optimization which may be crucial for further scaling.