mlsanityData to 7 Oct 2026

VoiceBench: Benchmarking LLM-Based Voice Assistants

arXiv 2410.17196 ↗First posted 22 Oct 2024cs.CL cs.AI cs.SDcited by 192

Not on the map. Why?

Cited by 22 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering a significantly improved user experience compared to traditional text-based interactions. However, the absence of benchmarks designed to evaluate these speech interaction capabilities has hindered progress of LLM-based voice assistants development. Current evaluations focus primarily on automatic speech recognition (ASR) or general knowledge evaluation with clean speeches, neglecting the more intricate, real-world scenarios that involve diverse speaker characteristics, environmental and content factors. To address this, we introduce VoiceBench, the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants. VoiceBench also includes both real and synthetic spoken instructions that incorporate the above three key real-world variations. Extensive experiments reveal the limitations of current LLM-based voice assistant models and offer valuable insights for future research and development in this field.

Key facts. Cited by 192 papers in the map (data through 7 Oct 2026). Builds on 50 of its 91 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

50 of 91 references.

References · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
38 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  3. 2024Qwen2-Audio Technical Reportcited by 543
  4. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  5. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  6. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  8. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  9. 2023GPTScore: Evaluate as You Desirecited by 357
  10. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  11. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  12. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  13. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  14. 2023Listen, Think, and Understandcited by 206
  15. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  16. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  17. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  18. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  19. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  20. 2018The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselinescited by 90
  21. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  22. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agentscited by 71
  23. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  24. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  25. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 36
  26. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  27. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  28. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 31
  29. 2024Resilience of Large Language Models for Noisy Instructionscited by 20
  30. 2020End-to-End Speech Recognition and Disfluency Removalcited by 16
  31. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialoguecited by 16
  32. 2024TTSlow: Slow Down Text-to-Speech with Efficiency Robustness Evaluationscited by 12
  33. 2023Dynamic Transformers Provide a False Sense of Efficiencycited by 11
  34. 2023A Chat About Boring Problems: Studying GPT-based text normalizationcited by 10
  35. 2023Automatic Pronunciation Assessment -- A Reviewcited by 7
  36. 2024Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Modelscited by 6
  37. 2024Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Modelscited by 4
  38. 2024CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 82; 2026: 107.

192 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2025Qwen3-Omni Technical Reportcited by 451
  3. 2025Kimi-Audio Technical Reportcited by 210
  4. 2026Qwen3.5-Omni Technical Reportcited by 146
  5. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  6. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  7. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  8. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  9. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  10. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  11. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 57
  12. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
38 more
  1. 2025Voxtralcited by 41
  2. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  3. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  4. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  5. 2025SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learningcited by 31
  6. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domainscited by 30
  7. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
  8. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 28
  9. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  10. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  11. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  12. 2025LongCat-Flash-Omni Technical Reportcited by 23
  13. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  14. 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 20
  15. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 20
  16. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  17. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
  18. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 17
  19. 2025Fun-Audio-Chat Technical Reportcited by 17
  20. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  21. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  22. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  23. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  24. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 15
  25. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  26. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  27. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 13
  28. 2025InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-trainingcited by 11
  29. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Modelscited by 10
  30. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 10
  31. 2025C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversationscited by 9
  32. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructionscited by 9
  33. 2025MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Modelscited by 9
  34. 2026Closing the Modality Reasoning Gap for Speech Large Language Modelscited by 9
  35. 2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillationcited by 9
  36. 2026ERNIE 5.0 Technical Reportcited by 9
  37. 2025Mind the Gap! Static and Interactive Evaluations of Large Audio Modelscited by 8
  38. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 8

50 most cited of 192 All 192 (JSON)