mlsanityData to 7 Oct 2026

Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation

arXiv 2607.18508 ↗First posted 20 Jul 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Preference over model-generated emotion descriptions is emerging as a standard evaluation metric for multimodal emotion understanding, exemplified by the MER2026 MER-Prefer track on EmoPrefer. Such benchmarks assume that predicting the preferred description requires grounded cross-modal understanding of the video. We conduct a systematic shortcut audit of EmoPrefer using content-blind probes. A simple logistic regression using only description length and generator identity, without processing the text, video, or audio, performs comparably to LoRA-finetuned 7B text and audio-visual judges (65.8 versus 66.8 WAF on EmoPrefer-V2). Generator identity is recoverable from description text with 99.5 percent accuracy, every candidate pair contrasts two distinct generators, and the human preference labels agree with a fold-exclusive per-generator win-rate prior on 66 percent of the evaluated pairs. When the human label conflicts with this prior, trained judges still follow the style prior on 63 to 80 percent of the pairs. On a length-matched subset that neutralizes verbosity bias, the tested media configurations yield no statistically significant improvement, while an ODIN-inspired diagnostic that decouples the style shortcut leaves its content head near chance. These results do not imply that human preferences are inherently stylistic or that the descriptions contain no emotional information. Instead, they show that the current scores can be reached without verifying either description against the video. We recommend source-balanced pairing, strict length control, counter-stereotypical sliced reporting, and multi-annotator consensus for future cross-generator evaluations. Code is available at https://github.com/jiabingyang01/EmoPrefer-Audit.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

18 of 32 references.

References · 18

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  10. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  12. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
6 more
  1. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  2. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  3. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  4. 2025Robust Reward Modeling via Causal Rubricscited by 18
  5. 2025MER 2025: When Affective Computing Meets Large Language Modelscited by 10
  6. 2025Rectifying Shortcut Behaviors in Preference-based Reward Learningcited by 6

Built on by

Not cited yet.