mlsanityData to 14 Oct 2026

When Does an ASR Error Matter? Task-Role Diagnostics for Tool-Using Voice Agents

arXiv 2610.17619 ↗First posted 13 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In cascaded voice agents, ASR errors can have markedly different downstream consequences: a small transcription error may leave a tool call unchanged or alter its intent or arguments. Conventional word error rate (WER), however, does not capture these differences. We introduce SpeechEvalX, a paired diagnostic protocol that evaluates how task roles affected by ASR errors influence structured tool calls. Controlled edits are matched by source utterance, edit count, and operation, while task-role labels are compared with lexical severity, BERT cosine distance, Semantic-WER, and binary criticality. Across 594 source utterances, joint call accuracy falls from 82.3% with reference transcripts to 71.9-74.9% across three ASR front ends. Number and negation edits cause 21.9-25.7 percentage-point losses, versus 1.4-2.9 points for control edits. Detailed task-role labels improve source-grouped ROC-AUC from 0.73 to 0.78 over binary criticality, showing that downstream impact depends strongly on the task role affected.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 28 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

25 of 28 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  10. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  11. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  12. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
13 more
  1. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  2. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  3. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  4. 2019A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understandingcited by 107
  5. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  6. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  7. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 42
  8. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domainscited by 36
  9. 2025VoiceAgentBench: Are Voice Assistants ready for agentic tasks?cited by 19
  10. 2026EVA-Bench: A New End-to-end Framework for Evaluating Voice Agentscited by 18
  11. 2021Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usabilitycited by 14
  12. 2025Measuring the Effect of Transcription Noise on Downstream Language Understanding Taskscited by 7
  13. 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agentscited by 5

Built on by

Not cited yet.