# Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

arXiv: https://arxiv.org/abs/2607.06827 · HTML: https://mlsanity.com/paper/2607.06827/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2026.

## Built on by (most cited first, 1 of 1)

- [Acoustic-to-Text KV Compression for Full-Duplex Speech Models](https://mlsanity.com/paper/2609.31224/) (2026)

## Abstract (arXiv)

Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.