# Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition

arXiv: https://arxiv.org/abs/2604.09121 · HTML: https://mlsanity.com/paper/2604.09121/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 23 of 23)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [Prompting Large Language Models with Speech Recognition Abilities](https://mlsanity.com/paper/2307.11795/) (2023)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [SLM: Bridge the thin gap between speech and text foundation models](https://mlsanity.com/paper/2310.00230/) (2023)
- [Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition](https://mlsanity.com/paper/2407.04675/) (2024)
- [FireRedASR: Open-Source Industrial-Grade Mandarin Speech Recognition Models from Encoder-Decoder to LLM Integration](https://mlsanity.com/paper/2501.14350/) (2025)
- [The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Results](https://mlsanity.com/paper/2007.05916/) (2020)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System](https://mlsanity.com/paper/2502.05512/) (2025)
- [Fun-ASR Technical Report](https://mlsanity.com/paper/2509.12508/) (2025)
- [Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usability](https://mlsanity.com/paper/2106.02016/) (2021)
- [LASER: An LLM-based ASR Scoring and Evaluation Rubric](https://mlsanity.com/paper/2510.07437/) (2025)
- [Denoising GER: A Noise-Robust Generative Error Correction with LLM for Speech Recognition](https://mlsanity.com/paper/2509.04392/) (2025)

## Built on by (most cited first, 1 of 1)

- [AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach](https://mlsanity.com/paper/2607.28175/) (2026)

## Abstract (arXiv)

Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale training data. However, two important aspects remain underexplored. First, Word Error Rate (WER), the dominant evaluation metric for decades, treats all words equally and often fails to reflect the semantic correctness of an utterance at the sentence level. Second, interactive correction-an essential component of human communication-has rarely been systematically studied in ASR research. In this paper, we integrate these two perspectives under an agentic framework for interactive ASR. We propose leveraging LLM-as-a-Judge as a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy. Furthermore, we design an LLM-driven agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs through semantic feedback. Extensive experiments are conducted on standard benchmarks, including GigaSpeech (English), WenetSpeech (Chinese), the ASRU 2019 code-switching test set. Both objective and subjective evaluations demonstrate the effectiveness of the proposed framework in improving semantic fidelity and interactive correction capability. We will release the code to facilitate future research in interactive and agentic ASR.