mlsanityData to 7 Oct 2026

PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

arXiv 2307.02762 ↗First posted 6 Jul 2023cs.CL cs.AIcited by 114

Not on the map. Why?

Cited by 26 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Nowadays, the quality of responses generated by different modern large language models (LLMs) is hard to evaluate and compare automatically. Recent studies suggest and predominantly use LLMs for reference-free evaluation of open-ended question answering. More specifically, they use the recognized "strongest" LLM as the evaluator, which conducts pairwise comparisons of candidate models' answers and provides a ranking score. However, this intuitive method has multiple problems, such as bringing in self-enhancement (favoring its own answers) and positional bias. We draw insights and lessons from the educational domain (Cho & MacArthur, 2011; Walsh, 2014) to improve LLM-based evaluations. Specifically, we propose (1) the peer rank (PR) algorithm that takes into account each peer LLM's pairwise preferences of all answer pairs, and outputs a final ranking of models; and (2) peer discussion (PD), where we prompt two LLMs to discuss and try to reach a mutual agreement on the preferences of two answers. We conduct experiments on two benchmark datasets. We find that our approaches achieve higher accuracy and align better with human judgments. Interestingly, PR can induce a relatively accurate self-ranking of models under the anonymous setting, where each model's name is unrevealed. Our work provides space to explore evaluating models that are hard to compare for humans.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

33 of 56 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
21 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  7. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  9. 2019ELI5: Long Form Question Answeringcited by 567
  10. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  12. 2023GPTScore: Evaluate as You Desirecited by 357
  13. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  14. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  15. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  16. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  17. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  18. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  19. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  20. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  21. 2014The PeerRank Method for Peer Assessmentcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 60; 2025: 25; 2026: 14.

114 papers.

Sort:

By area:

  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  5. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  6. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  8. 2023Large Language Model Alignment: A Surveycited by 241
  9. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  10. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  11. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  12. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
38 more
  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  2. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  3. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  4. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 92
  5. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  6. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 74
  7. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  8. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  9. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  10. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  11. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaborationcited by 48
  12. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  13. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  14. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  15. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debatecited by 40
  16. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 40
  17. 2023Debate Helps Supervise Unreliable Expertscited by 39
  18. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  19. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  20. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agentscited by 35
  21. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  22. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 33
  23. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 33
  24. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 32
  25. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 31
  26. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 30
  27. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agentscited by 30
  28. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  29. 2024Debate-to-Write: A Persona-Driven Multi-Agent Framework for Diverse Argument Generationcited by 29
  30. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
  31. 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
  32. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  33. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  34. 2024What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detectioncited by 24
  35. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 24
  36. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  37. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  38. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 22

50 most cited of 114 All 114 (JSON)