mlsanityData to 14 Oct 2026

A Survey on LLM-as-a-Judge

arXiv 2411.15594 ↗First posted 23 Nov 2024cs.CL cs.AIcited by 1,423

Not on the map. Why?

Cited by 289 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Accurate and consistent evaluation is crucial for decision-making across numerous fields, yet it remains a challenging task due to inherent subjectivity, variability, and scale. Large Language Models (LLMs) have achieved remarkable success across diverse domains, leading to the emergence of "LLM-as-a-Judge," where LLMs are employed as evaluators for complex tasks. With their ability to process diverse data types and provide scalable, cost-effective, and consistent assessments, LLMs present a compelling alternative to traditional expert-driven evaluations. However, ensuring the reliability of LLM-as-a-Judge systems remains a significant challenge that requires careful design and standardization. This paper provides a comprehensive survey of LLM-as-a-Judge, addressing the core question: How can reliable LLM-as-a-Judge systems be built? We explore strategies to enhance reliability, including improving consistency, mitigating biases, and adapting to diverse assessment scenarios. Additionally, we propose methodologies for evaluating the reliability of LLM-as-a-Judge systems, supported by a novel benchmark designed for this purpose. To advance the development and real-world deployment of LLM-as-a-Judge systems, we also discussed practical applications, challenges, and future directions. This survey serves as a foundational reference for researchers and practitioners in this rapidly evolving field.

Key facts. Cited by 1,423 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Nov 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 699; 2026: 715.

1,423 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  3. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  4. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  5. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
  6. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  7. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  8. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 141
  9. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 92
  10. 2025OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scalecited by 91
  11. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  12. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
38 more
  1. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  2. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  3. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  4. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 71
  5. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 71
  6. 2025Can Large Reasoning Models Self-Train?cited by 68
  7. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 64
  8. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  9. 2025Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agentscited by 62
  10. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 62
  11. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  12. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 60
  13. 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectoriescited by 60
  14. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 55
  15. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  16. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  17. 2025Understanding Aha Moments: from External Observations to Internal Mechanismscited by 50
  18. 2025Reward Reasoning Modelcited by 47
  19. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  20. 2025Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Usecited by 45
  21. 2025Correlated Errors in Large Language Modelscited by 45
  22. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 45
  23. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  24. 2025SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Informationcited by 42
  25. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  26. 2025MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verificationcited by 39
  27. 2025TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loopcited by 39
  28. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  29. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  30. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  31. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  32. 2025Optimizing Model Selection for Compound AI Systemscited by 36
  33. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 36
  34. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
  35. 2025MPMA: Preference Manipulation Attack Against Model Context Protocolcited by 34
  36. 2025Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoningcited by 34
  37. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  38. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 34

50 most cited of 1,423 All 1,423 (JSON)