ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Not on the map. Why?
Abstract (arXiv)
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 30 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
28 of 30 references.
References · 28
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2024GPT-4o System Cardcited by 4,588
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2025Qwen2.5-Omni Technical Reportcited by 736
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
16 more
- 2025Kimi-Audio Technical Reportcited by 222
- 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
- 2024EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberationcited by 64
- 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
- 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
- 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
- 2025ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interactioncited by 24
- 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 22
- 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 21
- 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 17
- 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
- 2025InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systemscited by 17
- 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
- 2025SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluationcited by 14
- 2024Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistantscited by 6
- 2026Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluationcited by 5
Built on by
Not cited yet.