# Acoustic-to-Text KV Compression for Full-Duplex Speech Models

arXiv: https://arxiv.org/abs/2609.31224 · HTML: https://mlsanity.com/paper/2609.31224/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 22 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction](https://mlsanity.com/paper/2604.27393/) (2026)
- [Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents](https://mlsanity.com/paper/2409.15594/) (2024)
- [Language Model Can Listen While Speaking](https://mlsanity.com/paper/2408.02622/) (2024)
- [OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation](https://mlsanity.com/paper/2410.17799/) (2024)
- [A Full-duplex Speech Dialogue Scheme Based On Large Language Models](https://mlsanity.com/paper/2405.19487/) (2024)
- [InfiniPot: Infinite Context Processing on Memory-Constrained LLMs](https://mlsanity.com/paper/2410.01518/) (2024)
- [SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models](https://mlsanity.com/paper/2510.06917/) (2025)
- [Can Speech LLMs Think while Listening?](https://mlsanity.com/paper/2510.07497/) (2025)
- [Chronological Thinking in Full-Duplex Spoken Dialogue Language Models](https://mlsanity.com/paper/2510.05150/) (2025)
- [The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning](https://mlsanity.com/paper/2603.17837/) (2026)
- [FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model](https://mlsanity.com/paper/2410.03007/) (2024)
- [LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech](https://mlsanity.com/paper/2601.13539/) (2026)
- [AudioKV: KV Cache Eviction in Efficient Large Audio Language Models](https://mlsanity.com/paper/2604.06694/) (2026)
- [Learning When to Think While Listening in Large Audio-Language Models](https://mlsanity.com/paper/2605.27190/) (2026)
- [Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models](https://mlsanity.com/paper/2602.05373/) (2026)
- [Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs](https://mlsanity.com/paper/2607.06827/) (2026)

## Abstract (arXiv)

Full-duplex speech language models continuously accumulate acoustic key-value (KV) states, making long-running interactions memory-intensive. During listening, the model can finish processing an audio unit before the next arrives; we term the remaining interval listening-time slack. We propose acoustic-to-text KV compression, which introduces a transcription side channel to convert incoming speech into compact textual memory within this interval. When the cache exceeds a target budget during inference, older acoustic states are evicted while transcripts and recent acoustic context remain. We train the side channel with LoRA using cross-entropy on transcription segments. To preserve listening and speaking behavior, we apply knowledge distillation to the original model's token-level output distributions at native prediction positions. On ten-minute LongSpeech sessions, our MiniCPM-o 4.5 implementation reduces peak streaming KV-cache size by 64.6% compared with the same model without eviction. The proposed method also improves transcription, temporal question answering, and summarization over the baseline. Full-Duplex-Bench evaluations further show comparable pause-handling, turn-taking, and interruption performance.