Same Words, Different Judgments: How Preferences Vary Across Modalities
Not on the map. Why?
Abstract (arXiv)
Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first ICC-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving textitgood agreement within either modality (ICC(2,k) ≈ .80) requires ~9 raters. At the same time, modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 26 Feb 2026.
Builds on
21 of 30 references.
References · 21
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2020Learning to summarize from human feedbackcited by 2,588
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
- 2024Qwen2-Audio Technical Reportcited by 560
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
- 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
- 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
- 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
9 more
- 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
- 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
- 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
- 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 47
- 2024Believing Anthropomorphism: Examining the Role of Anthropomorphic Cues on Trust in Large Language Modelscited by 37
- 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
- 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 32
- 2024Speechworthy Instruction-tuned Language Modelscited by 7
- 2025Preference-Based Learning in Audio Applications: A Systematic Analysiscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026RLHF May Not Reflect Genuine Preferences
- 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
- 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
- 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback
- 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
- 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
- 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
- 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Model