mlsanityData to 14 Oct 2026

From Lists to Emojis: How Format Bias Affects Model Alignment

arXiv 2409.11704 ↗First posted 18 Sep 2024cs.CL cs.LGcited by 49

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this paper, we study format biases in reinforcement learning from human feedback (RLHF). We observe that many widely-used preference models, including human evaluators, GPT-4, and top-ranking models on the RewardBench benchmark, exhibit strong biases towards specific format patterns, such as lists, links, bold text, and emojis. Furthermore, large language models (LLMs) can exploit these biases to achieve higher rankings on popular benchmarks like AlpacaEval and LMSYS Chatbot Arena. One notable example of this is verbosity bias, where current preference models favor longer responses that appear more comprehensive, even when their quality is equal to or lower than shorter, competing responses. However, format biases beyond verbosity remain largely underexplored in the literature. In this work, we extend the study of biases in preference learning beyond the commonly recognized length bias, offering a comprehensive analysis of a wider range of format biases. Additionally, we show that with a small amount of biased data (less than 1%), we can inject significant bias into the reward model. Moreover, these format biases can also be easily exploited by downstream alignment algorithms, such as best-of-n sampling and online iterative DPO, as it is usually easier to manipulate the format than to improve the quality of responses. Our findings emphasize the need to disentangle format and content both for designing alignment algorithms and evaluating models.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 73 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

53 of 73 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024BOND: Aligning LLMs with Best-of-N Distillation19 Jul
  2. 2024WARP: On the Benefits of Weight Averaged Rewarded Policies24 Jun
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  4. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  5. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  6. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  9. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  12. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
7 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Mistral 7Bcited by 3,164
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  12. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
22 more
  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  2. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  3. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  4. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  5. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  6. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  7. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  9. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  10. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  11. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  12. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  13. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  14. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  15. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  16. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  17. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  18. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  19. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  20. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54
  21. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  22. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 20; 2026: 24.

49 papers.

Sort:

By area:

  1. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 51
  4. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  5. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  6. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  7. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  8. 2025HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimizationcited by 21
  9. 2025Robust Reward Modeling via Causal Rubricscited by 19
  10. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
  11. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  12. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
37 more
  1. 2026Real-Time Aligned Reward Model beyond Semanticscited by 13
  2. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  3. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scorescited by 7
  4. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  5. 2025Preference Learning for AI Alignment: a Causal Perspectivecited by 6
  6. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  7. 2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignmentcited by 5
  8. 2025Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?cited by 5
  9. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
  10. 2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 5
  11. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimizationcited by 4
  12. 2025Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken Englishcited by 4
  13. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 4
  14. 2026Automatically Finding Reward Model Biasescited by 4
  15. 2026Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judgescited by 4
  16. 2026A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluationcited by 4
  17. 2025Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedbackcited by 2
  18. 2025Plug-and-Play Dramaturge: A Divide-and-Conquer Approach for Iterative Narrative Script Refinement via Collaborative LLM Agentscited by 2
  19. 2026compar:IA: The French Government's LLM arena to collect French-language human prompts and preference datacited by 2
  20. 2026AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writingcited by 2
  21. 2026Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrantcited by 2
  22. 2025Exploring the Secondary Risks of Large Language Modelscited by 1
  23. 2026DeMix: Debugging Training Data with Mixed Data Error Types by Investigating Influence Vectorscited by 1
  24. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 1
  25. 2026Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Alignercited by 1
  26. 2025League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Modelscited by 0
  27. 2026Debiasing Reward Models via Causally Motivated Inference-Time Interventioncited by 0
  28. 2026Configurable Reward Model for Balanced Safety Alignmentcited by 0
  29. 2026Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learningcited by 0
  30. 2026Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Modelscited by 0
  31. 2026What do Reward Models Memorize?cited by 0
  32. 2026UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifierscited by 0
  33. 2026Patterning in Practice: Debiasing Reward Models with Susceptibilitiescited by 0
  34. 2026BiasReducer: Adaptive Bias Mitigation for Reward Modelscited by 0
  35. 2026RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedbackcited by 0
  36. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignmentcited by 0
  37. 2026Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  2. 2025Detecting Prefix Bias in LLM-based Reward Models
  3. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  4. 2023Verbosity Bias in Preference Labeling by Large Language Models
  5. 2024REFINE-LM: Mitigating Language Model Stereotypes via Reinforcement Learning
  6. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  7. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling
  8. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHF