mlsanityData to 7 Oct 2026

Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking

arXiv 2409.15268 ↗First posted 23 Sep 2024cs.LG cs.AIcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior alignment by virtue of better correspondence with human pairwise preferences, often measured by LLM-judges. In this work, we attempt to answer the following question -- do LLM-judge preferences translate to progress on other, more concrete metrics for alignment, and if not, why not? We define a concrete metric for alignment, and introduce SOS-Bench (Substance Outweighs Style Benchmark), which is to the best of our knowledge the largest standardized, reproducible LLM meta-benchmark to date. We find that (1) LLM-judge preferences do not correlate with concrete measures of safety, world knowledge, and instruction following; (2) LLM-judges have powerful implicit biases, prioritizing style over factuality and safety; and (3) the supervised fine-tuning (SFT) stage of post-training, and not the PO stage, has the greatest impact on alignment, with data scaling and prompt diversity as the driving factors. Our codebase and complete results can be found at https://github.com/penfever/sos-bench.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 63 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

48 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates23 Aug
  2. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  3. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications24 Oct

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2023LIMA: Less Is More for Alignmentcited by 1,080
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
33 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  5. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  10. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  11. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  13. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  14. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  15. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  16. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  17. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  18. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  19. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  20. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  21. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  22. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  23. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  24. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  25. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  26. 2018Categorizing Variants of Goodhart's Lawcited by 132
  27. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  28. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 52
  29. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  30. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  31. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  32. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 25
  33. 2024WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 11.

35 papers.

Sort:

By area:

  1. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  2. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  3. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 39
  4. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  5. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  6. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  7. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  8. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  9. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  10. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidencecited by 12
  11. 2025WorldPM: Scaling Human Preference Modelingcited by 12
  12. 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validitycited by 11
23 more
  1. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  2. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  3. 2025Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level IIIcited by 6
  4. 2025Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferencescited by 6
  5. 2025Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structurescited by 5
  6. 2025WILDCHAT-50M: A Deep Dive Into the Role of Synthetic Data in Post-Trainingcited by 4
  7. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  8. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  9. 2026Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judgescited by 3
  10. 2026Position: Evaluation Scores Are Perishable Knowledge Claimscited by 3
  11. 2025Reward Models are Metrics in a Trench Coatcited by 2
  12. 2025Plug-and-Play Dramaturge: A Divide-and-Conquer Approach for Iterative Narrative Script Refinement via Collaborative LLM Agentscited by 2
  13. 2026TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-trainingcited by 2
  14. 2026Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboardscited by 2
  15. 2025A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokenscited by 1
  16. 2025RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competitioncited by 1
  17. 2026A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agentscited by 1
  18. 2026Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluationcited by 1
  19. 2026Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytellingcited by 1
  20. 2025IDEAlign: Comparing Ideas of Large Language Models to Domain Expertcited by 0
  21. 2026RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluationcited by 0
  22. 2026Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialoguecited by 0
  23. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dissecting Human and LLM Preferences
  2. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  3. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale
  4. 2026Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
  5. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  7. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment
  8. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment