# KV-Lingo: Learning KV-Cache Translators with Distillation

arXiv: https://arxiv.org/abs/2609.32610 · HTML: https://mlsanity.com/paper/2609.32610/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 33 of its 89 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 33)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [Gemma 4 Technical Report](https://mlsanity.com/paper/2607.02770/) (2026)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [Olmo 3](https://mlsanity.com/paper/2512.13961/) (2025)
- [Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2512.20848/) (2025)
- [No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization](https://mlsanity.com/paper/2402.18096/) (2024)
- [Don't be lazy: CompleteP enables compute-efficient deep transformers](https://mlsanity.com/paper/2505.01618/) (2025)
- [KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing](https://mlsanity.com/paper/2410.18517/) (2024)
- [DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving](https://mlsanity.com/paper/2411.02820/) (2024)
- [Latent Space Communication via K-V Cache Alignment](https://mlsanity.com/paper/2601.06123/) (2026)
- [Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse](https://mlsanity.com/paper/2608.03893/) (2026)
- [KV Prediction for Improved Time to First Token](https://mlsanity.com/paper/2410.08391/) (2024)
- [RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse](https://mlsanity.com/paper/2603.13289/) (2026)
- [See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents](https://mlsanity.com/paper/2606.13594/) (2026)
- [Training Transformers for KV Cache Compressibility](https://mlsanity.com/paper/2605.05971/) (2026)
- [CacheBridge: Efficient Cross-Model KV Cache Transfer](https://mlsanity.com/paper/2609.00891/) (2026)

## Built on by (most cited first, 1 of 1)

- [Emergent Structure in the Marginal Attention Space of Language Models](https://mlsanity.com/paper/2610.03109/) (2026)

## Abstract (arXiv)

Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatible representations. This makes it costly to switch models over a shared context: although the context has already been processed by one model, the incoming model must process it again to build its own cache. We introduce KV-Lingo, a method for translating the KV cache of a source model into one that can be read by a target model. KV-Lingo consists of a collection of linear maps, typically one per layer of the target model, that are applied independently on all tokens' key and value representations. We train these maps using distillation, minimising the divergence between the target model's predictions from its native cache and those from the translated cache. We consider several model pairs spanning multiple sizes and architectures, training one translator per pair on a generic text corpus. The resulting translators preserve strong downstream performance in both small-to-large and large-to-small transfers. Since a switch then costs a linear map and a single decoding step instead of a prefill, replacing re-prefill with cache translation reduces the time to first token after a model switch by 9.6x already on a 64-token prompt for Qwen models on an Apple M3 Ultra, and by up to 29x at 32k context length on an H100. These gains make KV-Lingo particularly useful for dynamic model routing: a context can be processed by one model and handed off to another only when needed, without re-prefilling the shared prefix. We finally show that KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in our multi-turn evaluations.