# SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant

arXiv: https://arxiv.org/abs/2506.02457 · HTML: https://mlsanity.com/paper/2506.02457/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2025.

## Built on by (most cited first, 7 of 7)

- [MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models](https://mlsanity.com/paper/2511.00850/) (2025)
- [VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models](https://mlsanity.com/paper/2601.19956/) (2026)
- [Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluation](https://mlsanity.com/paper/2601.13742/) (2026)
- [On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation](https://mlsanity.com/paper/2601.06329/) (2026)
- [WASIL: In-the-Wild Arabic Spoken Interactions with LLMs](https://mlsanity.com/paper/2605.16364/) (2026)
- [CallScreenBench: Benchmarking Small Language Models as Phone Secretaries](https://mlsanity.com/paper/2608.01033/) (2026)
- [S3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants](https://mlsanity.com/paper/2609.09852/) (2026)

## Abstract (arXiv)

Thanks to the steady progress of large language models (LLMs), speech encoding algorithms and vocoder structure, recent advancements have enabled generating speech response directly from a user instruction. However, benchmarking the generated speech quality has been a neglected but critical issue, considering the shift from the pursuit of semantic accuracy to vivid and spontaneous speech flow. Previous evaluation focused on the speech-understanding ability, lacking a quantification of acoustic quality. In this paper, we propose Speech cOnversational Voice Assistant Benchmark (SOVA-Bench), providing a comprehension comparison of the general knowledge, speech recognition and understanding, along with both semantic and acoustic generative ability between available speech LLMs. To the best of our knowledge, SOVA-Bench is one of the most systematic evaluation frameworks for speech LLMs, inspiring the direction of voice interaction systems.