mlsanityData to 7 Oct 2026

Large Language Models are not Fair Evaluators

arXiv 2305.17926 ↗First posted 29 May 2023cs.CL cs.AI cs.IRcited by 1,050

Not on the map. Why?

Cited by 129 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we uncover a systematic bias in the evaluation paradigm of adopting large language models~(LLMs), e.g., GPT-4, as a referee to score and compare the quality of responses generated by candidate models. We find that the quality ranking of candidate responses can be easily hacked by simply altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other, e.g., Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator. To address this issue, we propose a calibration framework with three simple yet effective strategies: 1) Multiple Evidence Calibration, which requires the evaluator model to generate multiple evaluation evidence before assigning ratings; 2) Balanced Position Calibration, which aggregates results across various orders to determine the final score; 3) Human-in-the-Loop Calibration, which introduces a balanced position diversity entropy to measure the difficulty of each example and seeks human assistance when needed. We also manually annotate the "win/tie/lose" outcomes of responses from ChatGPT and Vicuna-13B in the Vicuna Benchmark's question prompt, and extensive experiments demonstrate that our approach successfully mitigates evaluation bias, resulting in closer alignment with human judgments. We release our code and human annotation at urlhttps://github.com/i-Eval/FairEval to facilitate future research.

Key facts. Cited by 1,050 papers in the map (data through 7 Oct 2026). Builds on 26 of its 45 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

26 of 45 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  7. 2023LIMA: Less Is More for Alignmentcited by 1,080
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2023Instruction Tuning with GPT-4cited by 676
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
14 more
  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  3. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  4. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  5. 2023Eight Things to Know about Large Language Modelscited by 73
  6. 2019Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inferencecited by 48
  7. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  8. 2017The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Taskcited by 39
  9. 2020HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inferencecited by 17
  10. 2020An Empirical Study on Model-agnostic Debiasing Strategies for Robust Natural Language Inferencecited by 16
  11. 2022Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentationcited by 13
  12. 2021Behind the Scenes: An Exploration of Trigger Biases Problem in Few-Shot Event Classificationcited by 8
  13. 2023Enhancing Continual Relation Extraction via Classifier Decompositioncited by 8
  14. 2023RepCL: Exploring Effective Representation for Continual Text Classificationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 123; 2024: 267; 2025: 245; 2026: 415.

1,050 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  6. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  11. 2023The Falcon Series of Open Language Modelscited by 526
  12. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
38 more
  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  5. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  8. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  9. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  10. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  11. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  12. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  13. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  14. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  15. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  16. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  17. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  18. 2023Challenges and Applications of Large Language Modelscited by 251
  19. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  20. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  21. 2023Large Language Model Alignment: A Surveycited by 241
  22. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  23. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  24. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  25. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  26. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  27. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  28. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  29. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  30. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  31. 2023Generative Judge for Evaluating Alignmentcited by 163
  32. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  33. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  34. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  35. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  36. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  37. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  38. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142

50 most cited of 1,050 All 1,050 (JSON)