# EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

arXiv: https://arxiv.org/abs/2605.13841 · HTML: https://mlsanity.com/paper/2605.13841/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 34 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 8 of 8)

- [Mind the Sim2Real Gap in User Simulation for Agentic Tasks](https://mlsanity.com/paper/2603.11245/) (2026)
- [Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency](https://mlsanity.com/paper/2604.04847/) (2026)
- [Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation](https://mlsanity.com/paper/2603.03116/) (2026)
- [VoiceAgentBench: Are Voice Assistants ready for agentic tasks?](https://mlsanity.com/paper/2510.07978/) (2025)
- [Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems](https://mlsanity.com/paper/2511.14136/) (2025)
- [SyGra: A Unified Graph-Based Framework for Scalable Generation, Quality Tagging, and Management of Synthetic Data](https://mlsanity.com/paper/2508.15432/) (2025)
- [Designing Sequence with Minimum PSL Using Chebyshev Distance and its Application for Chaotic MIMO Radar Waveform Design](https://mlsanity.com/paper/2010.03674/) (2020)
- [Testing the Testers: Human-Driven Quality Assessment of Voice AI Testing Platforms](https://mlsanity.com/paper/2511.04133/) (2025)

## Built on by (most cited first, 18 of 18)

- [VAmoS Bench: Voice Agent Simulation Bench](https://mlsanity.com/paper/2607.27453/) (2026)
- [IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows](https://mlsanity.com/paper/2606.19595/) (2026)
- [MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents](https://mlsanity.com/paper/2609.20152/) (2026)
- [A frontend-backend architecture for tool calls in full-duplex speech models](https://mlsanity.com/paper/2609.19334/) (2026)
- [Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction](https://mlsanity.com/paper/2609.25176/) (2026)
- [Just A Rather Very Intelligent Spoken Agent](https://mlsanity.com/paper/2607.16610/) (2026)
- [FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India](https://mlsanity.com/paper/2608.06027/) (2026)
- [When Patients Cut In: Extending Clinical Conversational AI Safety to Interruptions](https://mlsanity.com/paper/2608.29241/) (2026)
- [Auditing Bias and Safety in Voice AI Customer Care](https://mlsanity.com/paper/2609.04206/) (2026)
- [τ-Elicitation: Benchmarking multi-turn entity extraction in voice agents](https://mlsanity.com/paper/2609.13602/) (2026)
- [Inquesto Score: A reliability Protocol For Voice Agents](https://mlsanity.com/paper/2609.30514/) (2026)
- [Trust and Task Completion in the World of Consumer AI Agents](https://mlsanity.com/paper/2609.33017/) (2026)
- [CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings](https://mlsanity.com/paper/2609.35645/) (2026)
- [τ-Multilingual: Benchmarking Voice Agents Across Languages](https://mlsanity.com/paper/2609.35820/) (2026)
- [VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation](https://mlsanity.com/paper/2609.38512/) (2026)
- [Talk2Agent: Benchmarking Voice Interfaces for Text Agents](https://mlsanity.com/paper/2609.38867/) (2026)
- [Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators](https://mlsanity.com/paper/2610.11015/) (2026)
- [When Does an ASR Error Matter? Task-Role Diagnostics for Tool-Using Voice Agents](https://mlsanity.com/paper/2610.17619/) (2026)

## Abstract (arXiv)

Voice agents are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses realistic conversation simulation and comprehensive voice-specific evaluation. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates dynamic bot-to-bot audio conversations with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy) and EVA-X (Experience). EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and multi-trial measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--passk gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean Delta up to 0.314). We release EVA-Bench under an open-source license.