Data to 14 Oct 2026

OffsetBias: Leveraging Debiased Data for Tuning Evaluators

arXiv 2407.06551 ↗First posted 9 Jul 2024cs.CLcited by 94

Not on the map. Why?

Cited by 51 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Employing Large Language Models (LLMs) to assess the quality of generated responses, such as prompting instruct-tuned models or fine-tuning judge models, has become a widely adopted evaluation method. It is also known that such evaluators are vulnerable to biases, such as favoring longer responses. While it is important to overcome this problem, the specifics of these biases remain under-explored. In this work, we qualitatively identify six types of biases inherent in various judge models. We propose EvalBiasBench as a meta-evaluation collection of hand-crafted test cases for each bias type. Additionally, we present de-biasing dataset construction methods and the associated preference dataset OffsetBias. Experimental results demonstrate that fine-tuning on our dataset significantly enhances the robustness of judge models against biases and improves performance across most evaluation scenarios. We release our datasets and the fine-tuned judge model to public.

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

22 of 34 references.

References · 22

2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
2023Large Language Models are not Fair Evaluatorscited by 1,085
2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701

10 more

2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 47; 2026: 31.

94 papers.

Sort:

By area:

2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 51
2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44

38 more

2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 43
2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
2025Inference-Time Reward Hacking in Large Language Modelscited by 34
2024Direct Judgement Preference Optimizationcited by 33
2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 32
2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 29
2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 28
2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 21
2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 20
2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
2025PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalizationcited by 15
2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
2025WorldPM: Scaling Human Preference Modelingcited by 13
2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 13
2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
2026Reward Modeling from Natural Language Human Feedbackcited by 11
2024Interpreting Language Reward Models via Contrastive Explanationscited by 9
2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
2025From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational Systemcited by 9
2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferencescited by 8
2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 8
2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applicationscited by 7
2025Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflowscited by 7
2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gapcited by 7
2026WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agentscited by 7
2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
2025Reward Models Identify Consistency, Not Causalitycited by 6
2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 6
2025Quantitative LLM Judgescited by 6

50 most cited of 94 All 94 (JSON)