mlsanityData to 14 Oct 2026

AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation

arXiv 2507.12705 ↗First posted 17 Jul 2025cs.CL cs.SD eess.AScited by 21

Not on the map. Why?

Abstract (arXiv)

Current speech evaluation suffers from two critical limitations: the need and difficulty of designing specialized systems targeting individual audio characteristics, and poor correlation between automatic evaluation methods and human preferences. This work presents a systematic study of Large Audio Model (LAM) as a Judge, AudioJudge, investigating whether it can provide a unified evaluation framework that addresses both challenges. We systematically explore AudioJudge across audio characteristic detection tasks, including pronunciation, speaking rate, speaker identification and speech quality, and system-level human preference simulation for automated benchmarking. We investigate different prompt engineering strategies, finding that audio concatenation combined with in-context learning significantly improves performance across both audio characteristic detection and human preference simulation tasks. We further introduce a multi-aspect ensemble AudioJudge to enable general-purpose multi-aspect audio evaluation. This method decomposes speech assessment into specialized judges for lexical content, speech quality, and paralinguistic features, achieving up to 0.91 Spearman correlation with human preferences on our system ranking benchmark. Robustness analysis reveals that while LAMs maintain strong performance under acoustic noise, they exhibit significant verbosity and positional biases that require careful mitigation.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

36 of 47 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  5. 2022A Survey on In-context Learningcited by 721
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  8. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  9. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  10. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  11. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  12. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
24 more
  1. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  2. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  3. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  4. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
  5. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 127
  6. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  7. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 72
  8. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 71
  9. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 68
  10. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  11. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
  12. 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 30
  13. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  14. 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 22
  15. 2024PAM: Prompting Audio-Language Models for Audio Quality Assessmentcited by 20
  16. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  17. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  18. 2023A Study on Incorporating Whisper for Robust Speech Assessmentcited by 11
  19. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
  20. 2024LLM-based speaker diarization correction: A generalizable approachcited by 8
  21. 2025Mind the Gap! Static and Interactive Evaluations of Large Audio Modelscited by 8
  22. 2021InQSS: a speech intelligibility and quality assessment model using a multi-task learning networkcited by 7
  23. 2021Weakly-supervised word-level pronunciation error detection in non-native English speechcited by 4
  24. 2022Text-to-speech synthesis from dark data with evaluation-in-the-loop data selectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 19.

21 papers.

Sort:

By area:

  1. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  2. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  3. 2026Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluationcited by 5
  4. 2026GSRM: Generative Speech Reward Model for Speech RLHFcited by 4
  5. 2025Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generationcited by 3
  6. 2026TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosiscited by 2
  7. 2026JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructionscited by 2
  8. 2026Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluationcited by 2
  9. 2026The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speechcited by 2
  10. 2026A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agentscited by 1
  11. 2026Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluationcited by 1
  12. 2026Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correctioncited by 1
9 more
  1. 2026Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignmentcited by 0
  2. 2026ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judgecited by 0
  3. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  4. 2026CallScreenBench: Benchmarking Small Language Models as Phone Secretariescited by 0
  5. 2026Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensionscited by 0
  6. 2026SteerDuplex: Steerable Duplex Speech Dialogue Modelscited by 0
  7. 2026NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speechcited by 0
  8. 2026SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferencescited by 0
  9. 2026What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratingscited by 0