# Multimodal Conversational Context for LLM-Based ASR: Data Construction, Training, and Benchmark

arXiv: https://arxiv.org/abs/2609.19765 · HTML: https://mlsanity.com/paper/2609.19765/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech recognition models and training, one of 11 directions in Speech and audio language models.
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Placed in

- Speech and audio language models / [Speech recognition models and training](https://mlsanity.com/speech/speech-recognition-models-and-training/)

## Builds on (most cited first, 21 of 21)

- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [Step-Audio 2 Technical Report](https://mlsanity.com/paper/2507.16632/) (2025)
- [Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition](https://mlsanity.com/paper/2407.04675/) (2024)
- [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025)
- [Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audio](https://mlsanity.com/paper/2308.05218/) (2023)
- [Target Speaker ASR with Whisper](https://mlsanity.com/paper/2409.09543/) (2024)
- [X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System](https://mlsanity.com/paper/2512.18706/) (2025)
- [DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization](https://mlsanity.com/paper/2603.09180/) (2026)
- [Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm](https://mlsanity.com/paper/2607.23938/) (2026)
- [Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation](https://mlsanity.com/paper/2310.14278/) (2023)
- [MOSS Transcribe Diarize Technical Report](https://mlsanity.com/paper/2601.01554/) (2026)
- [PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech](https://mlsanity.com/paper/2512.23686/) (2025)
- [Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning](https://mlsanity.com/paper/2408.06043/) (2024)
- [Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems](https://mlsanity.com/paper/2602.23266/) (2026)
- [Qwen-Audio-3.0-ASR Technical Report](https://mlsanity.com/paper/2609.07549/) (2026)
- [From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs](https://mlsanity.com/paper/2603.24034/) (2026)

## Abstract (arXiv)

Conversational context provides semantic and acoustic cues across turns for automatic speech recognition (ASR), but relying on historical transcripts can propagate recognition errors and discard pronunciation and speaker information. We present a multimodal conversational-context framework for LLM-based ASR that integrates a scenario-controlled data pipeline, scalable multimodal context training, and systematic evaluation. We construct dialogues around entities and their confusable forms and interleave historical user speech with assistant text responses for supervised fine-tuning. We also introduce MM-ContextASR Bench, which evaluates contextual understanding and entity error correction across five scenarios. Experiments with Qwen3-Omni and Step-Audio-2-mini reveal limitations in handling irrelevant and erroneous history and show that our data construction and training improve context utilization, with multimodal context achieving the highest overall entity recall on both models. Further experiments on accent, dialect, and target-speaker ASR demonstrate the value of historical speech. The benchmark data and evaluation code are publicly available at https://github.com/llh666521/MM-ContextASR.