# Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking

arXiv: https://arxiv.org/abs/2409.15268 · HTML: https://mlsanity.com/paper/2409.15268/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 48 of its 63 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 48)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)

## Built on by (most cited first, 30 of 35)

- [No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding](https://mlsanity.com/paper/2503.05061/) (2025)
- [Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators](https://mlsanity.com/paper/2504.15253/) (2025)
- [LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing](https://mlsanity.com/paper/2507.00769/) (2025)
- [Neither Valid nor Reliable? Investigating the Use of LLMs as Judges](https://mlsanity.com/paper/2508.18076/) (2025)
- [Do LLM Evaluators Prefer Themselves for a Reason?](https://mlsanity.com/paper/2504.03846/) (2025)
- [JuStRank: Benchmarking LLM Judges for System Ranking](https://mlsanity.com/paper/2412.09569/) (2024)
- [Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results](https://mlsanity.com/paper/2504.13677/) (2025)
- [A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness](https://mlsanity.com/paper/2603.06594/) (2026)
- [WorldPM: Scaling Human Preference Modeling](https://mlsanity.com/paper/2505.10527/) (2025)
- [Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models](https://mlsanity.com/paper/2506.05339/) (2025)
- [How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence](https://mlsanity.com/paper/2504.02904/) (2025)
- [When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity](https://mlsanity.com/paper/2509.20293/) (2025)
- [J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization](https://mlsanity.com/paper/2505.13346/) (2025)
- [SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat](https://mlsanity.com/paper/2506.04721/) (2025)
- [Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level III](https://mlsanity.com/paper/2507.02954/) (2025)
- [Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferences](https://mlsanity.com/paper/2511.02109/) (2025)
- [Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures](https://mlsanity.com/paper/2505.24069/) (2025)
- [WILDCHAT-50M: A Deep Dive Into the Role of Synthetic Data in Post-Training](https://mlsanity.com/paper/2501.18511/) (2025)
- [Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges](https://mlsanity.com/paper/2605.26156/) (2026)
- [Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector](https://mlsanity.com/paper/2509.07177/) (2025)
- [EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences](https://mlsanity.com/paper/2510.06370/) (2025)
- [Position: Evaluation Scores Are Perishable Knowledge Claims](https://mlsanity.com/paper/2607.26191/) (2026)
- [Reward Models are Metrics in a Trench Coat](https://mlsanity.com/paper/2510.03231/) (2025)
- [Plug-and-Play Dramaturge: A Divide-and-Conquer Approach for Iterative Narrative Script Refinement via Collaborative LLM Agents](https://mlsanity.com/paper/2510.05188/) (2025)
- [TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training](https://mlsanity.com/paper/2604.10784/) (2026)
- [Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards](https://mlsanity.com/paper/2604.21769/) (2026)
- [A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens](https://mlsanity.com/paper/2502.16366/) (2025)
- [RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition](https://mlsanity.com/paper/2506.14412/) (2025)
- [A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents](https://mlsanity.com/paper/2601.10809/) (2026)
- [Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation](https://mlsanity.com/paper/2603.05485/) (2026)

## Abstract (arXiv)

The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior alignment by virtue of better correspondence with human pairwise preferences, often measured by LLM-judges. In this work, we attempt to answer the following question -- do LLM-judge preferences translate to progress on other, more concrete metrics for alignment, and if not, why not? We define a concrete metric for alignment, and introduce SOS-Bench (Substance Outweighs Style Benchmark), which is to the best of our knowledge the largest standardized, reproducible LLM meta-benchmark to date. We find that (1) LLM-judge preferences do not correlate with concrete measures of safety, world knowledge, and instruction following; (2) LLM-judges have powerful implicit biases, prioritizing style over factuality and safety; and (3) the supervised fine-tuning (SFT) stage of post-training, and not the PO stage, has the greatest impact on alignment, with data scaling and prompt diversity as the driving factors. Our codebase and complete results can be found at https://github.com/penfever/sos-bench.