mlsanityData to 14 Oct 2026

Audio-Aware Large Language Models as Judges for Speaking Styles

arXiv 2506.05984 ↗First posted 6 Jun 2025eess.AS cs.AI cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Audio-aware large language models (ALLMs) can understand the textual and non-textual information in the audio input. In this paper, we explore using ALLMs as an automatic judge to assess the speaking styles of speeches. We use ALLM judges to evaluate the speeches generated by SLMs on two tasks: voice style instruction following and role-playing. The speaking style we consider includes emotion, volume, speaking pace, word emphasis, pitch control, and non-verbal elements. We use four spoken language models (SLMs) to complete the two tasks and use humans and ALLMs to judge the SLMs' responses. We compare two ALLM judges, GPT-4o-audio and Gemini-2.5-pro, with human evaluation results and show that the agreement between Gemini and human judges is comparable to the agreement between human evaluators. These promising results show that ALLMs can be used as a judge to evaluate SLMs. Our results also reveal that current SLMs, even GPT-4o-audio, still have room for improvement in controlling the speaking style and generating natural dialogues.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

18 of 34 references.

References · 18

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  7. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  8. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  9. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  10. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  11. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  12. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
6 more
  1. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  2. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  3. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  4. 2025S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Modelscited by 16
  5. 2023Why We Should Report the Details in Subjective Evaluation of TTS More Rigorouslycited by 10
  6. 2023Towards General-Purpose Text-Instruction-Guided Voice Conversioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 13.

17 papers.

Sort:

By area:

  1. 2025Game-Time: Evaluating Temporal Dynamics in Spoken Language Modelscited by 13
  2. 2025VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructionscited by 9
  3. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  4. 2026Hearing Between the Lines: Unlocking the Reasoning Power of LLMs for Speech Evaluationcited by 5
  5. 2025UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Modelscited by 2
  6. 2026Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accentscited by 2
  7. 2026Investigation for Relative Voice Impression Estimationcited by 1
  8. 2026Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Modelscited by 1
  9. 2026STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelitycited by 1
  10. 2026StanceBench: A Benchmark for Audio LLM-Based Interpersonal Stance Evaluation from Speechcited by 1
  11. 2026Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluationcited by 1
  12. 2025Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Modelscited by 0
5 more
  1. 2026Perfecting Human-AI Interaction at Clinical Scale. Turning Production Signals into Safer, More Human Conversationscited by 0
  2. 2026ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judgecited by 0
  3. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  4. 2026Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensionscited by 0
  5. 2026Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidancecited by 0