# Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments

arXiv: https://arxiv.org/abs/2406.11370 · HTML: https://mlsanity.com/paper/2406.11370/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering](https://mlsanity.com/paper/2309.17249/) (2023)
- [Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study](https://mlsanity.com/paper/2304.00723/) (2023)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [Better Zero-Shot Reasoning with Self-Adaptive Prompting](https://mlsanity.com/paper/2305.14106/) (2023)
- [Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations](https://mlsanity.com/paper/2305.15035/) (2023)

## Built on by (most cited first, 25 of 25)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies](https://mlsanity.com/paper/2502.02533/) (2025)
- [No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding](https://mlsanity.com/paper/2503.05061/) (2025)
- [Self-Supervised Prompt Optimization](https://mlsanity.com/paper/2502.06855/) (2025)
- [Investigating Non-Transitivity in LLM-as-a-Judge](https://mlsanity.com/paper/2502.14074/) (2025)
- [Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction](https://mlsanity.com/paper/2509.18658/) (2025)
- [reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs](https://mlsanity.com/paper/2503.11751/) (2025)
- [From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation](https://mlsanity.com/paper/2502.00330/) (2025)
- [An Empirical Analysis of Uncertainty in Large Language Model Evaluations](https://mlsanity.com/paper/2502.10709/) (2025)
- [Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration](https://mlsanity.com/paper/2509.10704/) (2025)
- [Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models](https://mlsanity.com/paper/2410.02205/) (2024)
- [Tuning LLM Judge Design Decisions for 1/1000 of the Cost](https://mlsanity.com/paper/2501.17178/) (2025)
- [Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes](https://mlsanity.com/paper/2503.17755/) (2025)
- [Understanding Likelihood Over-optimisation in Direct Alignment Algorithms](https://mlsanity.com/paper/2410.11677/) (2024)
- [From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges](https://mlsanity.com/paper/2601.08654/) (2026)
- [MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization](https://mlsanity.com/paper/2510.01659/) (2025)
- [Agentic Policy Optimization via Instruction-Policy Co-Evolution](https://mlsanity.com/paper/2512.01945/) (2025)
- [SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization](https://mlsanity.com/paper/2510.21847/) (2025)
- [HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation](https://mlsanity.com/paper/2504.07174/) (2025)
- [UPA: Unsupervised Prompt Agent via Tree-Based Search and Selection](https://mlsanity.com/paper/2601.23273/) (2026)
- [SPEAR: Code-Augmented Agentic Prompt Optimization](https://mlsanity.com/paper/2605.26275/) (2026)
- [Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus](https://mlsanity.com/paper/2609.22512/) (2026)
- [COGNAC at SemEval-2026 Task 5: LLM Ensembles for Human-Level Word Sense Plausibility Rating in Challenging Narratives](https://mlsanity.com/paper/2603.15897/) (2026)
- [Multiple latent orderings better predict language model preferences](https://mlsanity.com/paper/2609.22170/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown promising abilities as cost-effective and reference-free evaluators for assessing language generation quality. In particular, pairwise LLM evaluators, which compare two generated texts and determine the preferred one, have been employed in a wide range of applications. However, LLMs exhibit preference biases and worrying sensitivity to prompt designs. In this work, we first reveal that the predictive preference of LLMs can be highly brittle and skewed, even with semantically equivalent instructions. We find that fairer predictive preferences from LLMs consistently lead to judgments that are better aligned with humans. Motivated by this phenomenon, we propose an automatic Zero-shot Evaluation-oriented Prompt Optimization framework, ZEPO, which aims to produce fairer preference decisions and improve the alignment of LLM evaluators with human judgments. To this end, we propose a zero-shot learning objective based on the preference decision fairness. ZEPO demonstrates substantial performance improvements over state-of-the-art LLM evaluators, without requiring labeled data, on representative meta-evaluation benchmarks. Our findings underscore the critical correlation between preference fairness and human alignment, positioning ZEPO as an efficient prompt optimizer for bridging the gap between LLM evaluators and human judgments.