# Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation

arXiv: https://arxiv.org/abs/2504.14716 · HTML: https://mlsanity.com/paper/2504.14716/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 20 Apr 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Disentangling Length from Quality in Direct Preference Optimization](https://mlsanity.com/paper/2403.19159/) (2024)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Language Models Learn to Mislead Humans via RLHF](https://mlsanity.com/paper/2409.12822/) (2024)
- [Verbosity Bias in Preference Labeling by Large Language Models](https://mlsanity.com/paper/2310.10076/) (2023)
- [Style Over Substance: Evaluation Biases for Large Language Models](https://mlsanity.com/paper/2307.03025/) (2023)
- [Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models](https://mlsanity.com/paper/2308.15812/) (2023)
- [Measuring the Inconsistency of Large Language Models in Preferential Ranking](https://mlsanity.com/paper/2410.08851/) (2024)
- [Using Natural Language Explanations to Rescale Human Judgments](https://mlsanity.com/paper/2305.14770/) (2023)

## Built on by (most cited first, 24 of 24)

- [Evaluating Scoring Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2506.22316/) (2025)
- [SCOPE: Selective Conformal Optimized Pairwise LLM Judging](https://mlsanity.com/paper/2602.13110/) (2026)
- [Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization](https://mlsanity.com/paper/2603.08091/) (2026)
- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling](https://mlsanity.com/paper/2604.08178/) (2026)
- [Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generation](https://mlsanity.com/paper/2605.26958/) (2026)
- [Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge](https://mlsanity.com/paper/2602.02219/) (2026)
- [FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge](https://mlsanity.com/paper/2602.06625/) (2026)
- [JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment](https://mlsanity.com/paper/2605.25240/) (2026)
- [Reference-Free Rating of LLM Responses via Latent Information](https://mlsanity.com/paper/2509.24678/) (2025)
- [Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation](https://mlsanity.com/paper/2604.10741/) (2026)
- [On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization](https://mlsanity.com/paper/2509.23542/) (2025)
- [Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?](https://mlsanity.com/paper/2510.00766/) (2025)
- [Offline A/B Testing of Slate Recommendation Systems with LLMs: Reducing the Dependency on Pre-Collected User Interaction Data](https://mlsanity.com/paper/2511.04541/) (2025)
- [Estimating Item Difficulty with Large Language Models as Experts](https://mlsanity.com/paper/2605.18562/) (2026)
- [Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines](https://mlsanity.com/paper/2605.26442/) (2026)
- [Scalable Evaluation of the Realism of Synthetic Environmental Augmentations in Images](https://mlsanity.com/paper/2603.04325/) (2026)
- [Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation](https://mlsanity.com/paper/2605.19776/) (2026)
- [Agent Step Value: Auditing Evaluation-Channel Sensitivity in Black-Box Agent Traces](https://mlsanity.com/paper/2607.04419/) (2026)
- [The Authority Expectancy Effect in Multi-Party Conflict](https://mlsanity.com/paper/2608.08026/) (2026)
- [The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability](https://mlsanity.com/paper/2609.07162/) (2026)
- [PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators](https://mlsanity.com/paper/2609.36086/) (2026)
- [MatrixReward: Reward from Rubric Matrix for Open-Ended Generation](https://mlsanity.com/paper/2610.00389/) (2026)
- [The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express](https://mlsanity.com/paper/2610.17158/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are widely used as proxies for human labelers in both training (Reinforcement Learning from AI Feedback) and large-scale response evaluation (LLM-as-a-judge). Alignment and evaluation are critical components in the development of reliable LLMs, and the choice of feedback protocol plays a central role in both but remains understudied. In this work, we show that the choice of feedback protocol for evaluation (absolute scores versus relative preferences) can significantly affect evaluation reliability and induce systematic biases. In the context of LLM-as-a-judge evaluation, we show that pairwise protocols are more vulnerable to distracted evaluation. Generator models can exploit spurious attributes (or distractor features) favored by the LLM judge, resulting in inflated scores for lower-quality outputs. We find that absolute scoring is more robust to such manipulation, producing judgments that better reflect response quality and are less influenced by distractor features. Our results demonstrate that generator models can flip preferences by embedding distractor features, skewing LLM-as-a-judge comparisons and leading to inaccurate conclusions about model quality in benchmark evaluations. Pairwise preferences flip in about 35% of the cases, compared to only 9% for absolute scores. We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.