mlsanityData to 14 Oct 2026

Auditing generative audio calls for known-task audio-llm evaluation

arXiv 2608.27817 ↗First posted 28 Aug 2026cs.SD cs.CL eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Speech and audio LLMs are evaluated by comparing waveform predictions with predictions from an automatic speech recognition (ASR) transcript. For fixed closed-set tasks, this conflates acoustic evidence with the need to invoke a generative audio model. We estimate incremental call value with matched selectors sharing pre-call evidence. Each policy may retain the transcript label, use a local encoder, or invoke a generative model; matched control removes generative actions but preserves pre-call evidence and development selection. On VocalSound, transcript-only accuracy is 0.296, while supervised CLAP and WavLM controls reach 0.850 and 0.854 without calls. Full selector reaches 0.925 at 12.5% calls versus 0.921 for matched No-call selector (difference 0.004; 95% CI [-0.025, 0.033]). Thus, results do not show a call gain after transcript and encoder evidence are available. Relevant quantity is incremental accuracy from allowing calls, not the waveform-transcript gap.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 31 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

29 of 31 references.

References · 29

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2025Qwen2.5-Omni Technical Reportcited by 736
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2021AST: Audio Spectrogram Transformercited by 643
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
17 more
  1. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  2. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  3. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  4. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  5. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  6. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  7. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 115
  8. 2025Step-Audio 2 Technical Reportcited by 105
  9. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  10. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  11. 2024A Survey on Speech Large Language Models for Understandingcited by 71
  12. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  13. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 19
  14. 2021An evaluation of word-level confidence estimation for end-to-end automatic speech recognitioncited by 18
  15. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialoguecited by 16
  16. 2026HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Modelscited by 8
  17. 2026The SLT 2026 SmartGlasses Challenge: Benchmarking Egocentric Multi-Talker Speech Recognition and Understanding with Audio-Language Modelscited by 1

Built on by

Not cited yet.