# VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

arXiv: https://arxiv.org/abs/2510.11098 · HTML: https://mlsanity.com/paper/2510.11098/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://mlsanity.com/paper/2412.02612/) (2024)
- [AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://mlsanity.com/paper/2402.07729/) (2024)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [OmniBench: Towards The Future of Universal Omni-Language Models](https://mlsanity.com/paper/2409.15272/) (2024)
- [Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM](https://mlsanity.com/paper/2305.15255/) (2023)
- [Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction](https://mlsanity.com/paper/2502.11946/) (2025)
- [Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction](https://mlsanity.com/paper/2502.17239/) (2025)
- [MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence](https://mlsanity.com/paper/2508.13992/) (2025)
- [Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models](https://mlsanity.com/paper/2507.23159/) (2025)
- [SD-QA: Spoken Dialectal Question Answering for the Real World](https://mlsanity.com/paper/2109.12072/) (2021)

## Built on by (most cited first, 4 of 4)

- [Covo-Audio Technical Report](https://mlsanity.com/paper/2602.09823/) (2026)
- [DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents](https://mlsanity.com/paper/2609.03423/) (2026)
- [VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context](https://mlsanity.com/paper/2511.08230/) (2025)
- [Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Models](https://mlsanity.com/paper/2609.36577/) (2026)

## Abstract (arXiv)

Recent advances in large audio language models (LALMs) have greatly enhanced multimodal conversational systems. However, existing benchmarks remain limited -- they are mainly English-centric, rely on synthetic speech, and lack comprehensive, discriminative evaluation across multiple dimensions. To address these gaps, we present Voice Chat Bot Bench (VCB Bench) -- a high-quality Chinese benchmark built entirely on real human speech. VCB Bench evaluates LALMs from three complementary perspectives: instruction following (including speech-level control beyond text commands), knowledge understanding (general knowledge, reasoning, and daily dialogue), and robustness (stability under perturbations in content, environment, and speaker traits). Experiments on representative LALMs reveal notable performance gaps and highlight future directions for improvement. VCB Bench provides a reproducible and fine-grained evaluation framework, offering standardized methodology and practical insights for advancing Chinese voice conversational models.