mlsanityData to 7 Oct 2026

Spoken Conversational Agents with Large Language Models

arXiv 2512.02593 ↗First posted 2 Dec 2025cs.CL cs.MA cs.NEcited by 4

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We frame adaptation of text LLMs to audio, cross-modal alignment, and joint speech-text training; review datasets, metrics, and robustness across accents and compare design choices (cascaded vs. E2E, post-ASR correction, streaming). We link industrial assistants to current open-domain and task-oriented agents, highlight reproducible baselines, and outline open problems in privacy, safety, and evaluation. Attendees leave with practical recipes and a clear systems-level roadmap.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 28 of its 44 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

28 of 44 references.

References · 28

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  5. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  6. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  8. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  9. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  10. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  11. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  12. 2018Building a Conversational Agent Overnight with Dialogue Self-Playcited by 70
16 more
  1. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  2. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
  3. 2017Towards Zero-Shot Frame Semantic Parsing for Domain Scalingcited by 40
  4. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  5. 2017End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learningcited by 19
  6. 2023End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directionscited by 17
  7. 2015Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguationcited by 13
  8. 2024gTBLS: Generating Tables from Text by Conditional Question Answeringcited by 13
  9. 2024Dense Passage Retrieval: Is it Retrieving?cited by 12
  10. 2023How to Estimate Model Transferability of Pre-Trained Speech Models?cited by 10
  11. 2024cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Paperscited by 10
  12. 2020mForms : Multimodal Form-Filling with Question Answeringcited by 5
  13. 2024Are Human Conversations Special? A Large Language Model Perspectivecited by 5
  14. 2016Is spoken language all-or-nothing? Implications for future speech-based human-machine interactioncited by 4
  15. 2024iTBLS: A Dataset of Interactive Conversations Over Tabular Informationcited by 3
  16. 2024Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026The Interspeech 2026 Audio Reasoning Challenge: Evaluating Reasoning Process Quality for Audio Reasoning Models and Agentscited by 6
  2. 2026DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolutioncited by 1
  3. 2026A2-LLM: An End-to-end Conversational Audio Avatar Large Language Modelcited by 0
  4. 2026Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
  2. 2026SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
  3. 2023Enhancing Pipeline-Based Conversational Agents with Large Language Models
  4. 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents
  5. 2026CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents
  6. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
  7. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
  8. 2026Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations