mlsanityData to 14 Oct 2026

From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge

arXiv 2411.16594 ↗First posted 25 Nov 2024cs.AI cs.CLcited by 494

Not on the map. Why?

Cited by 88 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Assessment and evaluation have long been critical challenges in artificial intelligence (AI) and natural language processing (NLP). Traditional methods, usually matching-based or small model-based, often fall short in open-ended and dynamic scenarios. Recent advancements in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs are leveraged to perform scoring, ranking, or selection for various machine learning evaluation scenarios. This paper presents a comprehensive survey of LLM-based judgment and assessment, offering an in-depth overview to review this evolving field. We first provide the definition from both input and output perspectives. Then we introduce a systematic taxonomy to explore LLM-as-a-judge along three dimensions: what to judge, how to judge, and how to benchmark. Finally, we also highlight key challenges and promising future directions for this emerging area. More resources on LLM-as-a-judge are on the website: https://llm-as-a-judge.github.io and https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge.

Key facts. Cited by 494 papers in the map (data through 14 Oct 2026). Builds on 190 of its 360 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

190 of 360 references.

References · 190

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  11. 2023Let's Verify Step by Stepcited by 3,789
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  13. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  14. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  15. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  16. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  17. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  18. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  19. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  20. 2022A Survey on In-context Learningcited by 721
  21. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  22. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  23. 2022Large Language Models Can Self-Improvecited by 664
  24. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  25. 2024Large Language Models: A Surveycited by 525
  26. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  27. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  28. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  29. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  30. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  31. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  32. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  33. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  34. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  35. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  36. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  37. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  38. 2023Data Selection for Language Models via Importance Resamplingcited by 329

50 most cited of 190 All 190 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 226; 2026: 257.

494 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
  4. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  5. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
  6. 2025SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Modelscited by 66
  7. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 60
  8. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 55
  9. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  10. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  11. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 54
  12. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
38 more
  1. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 44
  2. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  3. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  4. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  5. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 40
  6. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  7. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  8. 2025Self-Supervised Prompt Optimizationcited by 37
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  10. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  11. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
  12. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  13. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  14. 2025InSTA: Towards Internet-Scale Training For Agentscited by 32
  15. 2025ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquirycited by 32
  16. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 31
  17. 2025TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Reviewcited by 31
  18. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  19. 2025DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Modelscited by 30
  20. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 30
  21. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 30
  22. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
  23. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 28
  24. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  25. 2024Provable Scaling Laws for the Test-Time Compute of Large Language Modelscited by 26
  26. 2025CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulationcited by 26
  27. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  28. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 26
  29. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 24
  30. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 23
  31. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23
  32. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  33. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  34. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  35. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 21
  36. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  37. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 20
  38. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 19

50 most cited of 494 All 494 (JSON)