# Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR

arXiv: https://arxiv.org/abs/2603.26246 · HTML: https://mlsanity.com/paper/2603.26246/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Dialog-context aware end-to-end speech recognition](https://mlsanity.com/paper/1808.02171/) (2018)
- [Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers](https://mlsanity.com/paper/2104.09426/) (2021)
- [Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion](https://mlsanity.com/paper/1906.11604/) (2019)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [Text-only adaptation in LLM-based ASR through text denoising](https://mlsanity.com/paper/2601.20900/) (2026)
- [ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs](https://mlsanity.com/paper/2602.03226/) (2026)
- [Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning](https://mlsanity.com/paper/2408.06043/) (2024)
- [Post-decoder Biasing for End-to-End Speech Recognition of Multi-turn Medical Interview](https://mlsanity.com/paper/2403.00370/) (2024)

## Built on by (most cited first, 2 of 2)

- [How to Leverage Synthetic Speech for LLM-Based ASR Systems?](https://mlsanity.com/paper/2606.29031/) (2026)
- [When Synthetic Speech Is All You Have: Better Call GRPO](https://mlsanity.com/paper/2607.08409/) (2026)

## Abstract (arXiv)

Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.