mlsanityData to 7 Oct 2026

VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context

arXiv 2511.08230 ↗First posted 11 Nov 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The development of multi-modal large language models (LLMs) leads to intelligent approaches capable of speech interactions. As one of the most widely spoken languages globally, Mandarin is supported by most models to enhance their applicability and reach. However, the scarcity of comprehensive speech-to-speech (S2S) benchmarks in Mandarin contexts impedes systematic evaluation for developers and hinders fair model comparison for users. In this work, we propose VocalBench-zh, an ability-level divided evaluation suite adapted to Mandarin context consisting of 10 well-crafted subsets and over 10K high-quality instances, covering 12 user-oriented characters. The evaluation experiment on 14 mainstream models reveals the common challenges for current routes, and highlights the need for new insights into next-generation speech interactive systems. The evaluation codes and datasets will be available at https://github.com/SJTU-OmniAgent/VocalBench-zh.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 33 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

26 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025CS3-Bench: Evaluating and Enhancing Speech-to-Speech LLMs for Mandarin-English Code-Switching9 Oct
  2. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interaction10 Jan

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  8. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  9. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  10. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  11. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  12. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
12 more
  1. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  2. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  3. 2016Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answeringcited by 35
  4. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  5. 2022A Corpus for Understanding and Generating Moral Storiescited by 16
  6. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  7. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  8. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
  9. 2025InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-trainingcited by 11
  10. 2025VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewingcited by 7
  11. 2022CC-Riddle: A Question Answering Dataset of Chinese Character Riddlescited by 6
  12. 2025VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agentscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
  2. 2025WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild
  3. 2025VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluency
  4. 2026S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants
  5. 2025Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan
  6. 2026KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs
  7. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models
  8. 2026SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia