mlsanityData to 7 Oct 2026

LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

arXiv 2412.05579 ↗First posted 7 Dec 2024cs.CL cs.IRcited by 451

Not on the map. Why?

Cited by 79 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid advancement of Large Language Models (LLMs) has driven their expanding application across various fields. One of the most promising applications is their role as evaluators based on natural language responses, referred to as ''LLMs-as-judges''. This framework has attracted growing attention from both academia and industry due to their excellent effectiveness, ability to generalize across tasks, and interpretability in the form of natural language. This paper presents a comprehensive survey of the LLMs-as-judges paradigm from five key perspectives: Functionality, Methodology, Applications, Meta-evaluation, and Limitations. We begin by providing a systematic definition of LLMs-as-Judges and introduce their functionality (Why use LLM judges?). Then we address methodology to construct an evaluation system with LLMs (How to use LLM judges?). Additionally, we investigate the potential domains for their application (Where to use LLM judges?) and discuss methods for evaluating them in various contexts (How to evaluate LLM judges?). Finally, we provide a detailed analysis of the limitations of LLM judges and discuss potential future directions. Through a structured and comprehensive analysis, we aim aims to provide insights on the development and application of LLMs-as-judges in both research and practice. We will continue to maintain the relevant resource list at https://github.com/CSHaitao/Awesome-LLMs-as-Judges.

Key facts. Cited by 451 papers in the map (data through 7 Oct 2026). Builds on 222 of its 305 references that are in the map. First posted on arXiv 7 Dec 2024.

Builds on

222 of 305 references.

References · 222

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  11. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
38 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  13. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  14. 2022A Survey on In-context Learningcited by 717
  15. 2024Self-Rewarding Language Modelscited by 654
  16. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  17. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  18. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  19. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  20. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  21. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  22. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  23. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  24. 2023GPTScore: Evaluate as You Desirecited by 357
  25. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  26. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  27. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  28. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  29. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  30. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  31. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  32. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  33. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  34. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  35. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  36. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  37. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  38. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255

50 most cited of 222 All 222 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 226; 2026: 224.

451 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  3. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 132
  4. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  5. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  6. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  7. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  8. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 56
  9. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 56
  10. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 53
  11. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  12. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
38 more
  1. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  2. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  3. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 37
  4. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  5. 2025Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Modelscited by 33
  6. 2025ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewardingcited by 32
  7. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  8. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Modelscited by 32
  9. 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agentscited by 30
  10. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 27
  11. 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 26
  12. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  13. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 24
  14. 2025Normative Evaluation of Large Language Models with Everyday Moral Dilemmascited by 23
  15. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  16. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  17. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  18. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 22
  19. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  20. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 20
  21. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
  22. 2025LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversationcited by 19
  23. 2025The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaborationcited by 19
  24. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 19
  25. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  26. 2025Challenging the Evaluator: LLM Sycophancy Under User Rebuttalcited by 18
  27. 2025Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivitycited by 17
  28. 2025MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLMcited by 17
  29. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 17
  30. 2025Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solutioncited by 17
  31. 2026Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agentscited by 17
  32. 2025Unmasking Conversational Bias in AI Multiagent Systemscited by 16
  33. 2025ARGUS: Hallucination and Omission Evaluation in Video-LLMscited by 16
  34. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
  35. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 16
  36. 2026Intelligent AI Delegationcited by 16
  37. 2025Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generationcited by 15
  38. 2025Language Models' Factuality Depends on the Language of Inquirycited by 14

50 most cited of 451 All 451 (JSON)