Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation
Not on the map. Why?
Abstract (arXiv)
Large audio-language models (LALMs) are increasingly used as automatic judges for speech evaluation. However, high agreement with human ratings does not guarantee that their verdicts are grounded in the audio. A judge may instead rely on specialist labels or reference data supplied by the evaluation protocol itself, taking a shortcut in place of listening to the audio. In this paper, we audit such protocol-level ``shortcuts'' in LALM judges across three common deployment protocols: feature-blueprint judging, where the audio is replaced by a structured text description of acoustic features, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, we find that several LALMs rely on protocol-level shortcuts. For example, in feature-blueprint judging, incorrect specialist labels reduce five judges' emotion accuracy to 0.10 or below, and in concatenated A/B comparisons, Qwen3-Omni-Thinking often picks the same slot regardless of order swaps. These results indicate that aggregate agreement can overstate the validity of LALM judges unless the model and the evaluation protocol are assessed jointly, and that each model-protocol pair should be evaluated with a matched shortcut probe.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 15 Jul 2026.
Builds on
30 of 34 references.
References · 30
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2025Qwen2.5-Omni Technical Reportcited by 736
- 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
- 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
- 2024Qwen2-Audio Technical Reportcited by 560
- 2025Qwen3-Omni Technical Reportcited by 497
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
- 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
- 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
18 more
- 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
- 2025SycEval: Evaluating LLM Sycophancycited by 108
- 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
- 2022The VoiceMOS Challenge 2022cited by 57
- 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
- 2025Voxtralcited by 43
- 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
- 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
- 2025Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliancecited by 24
- 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 21
- 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systemscited by 17
- 2025SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluationcited by 14
- 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 8
- 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadoxcited by 7
- 2024CLAIR-A: Leveraging Large Language Models to Judge Audio Captionscited by 5
- 2026Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluationcited by 5
- 2026When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitrationcited by 5
- 2026LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialoguescited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.