mlsanityData to 14 Oct 2026

Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge

arXiv 2502.12501 ↗First posted 18 Feb 2025cs.CLcited by 19

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become a widely adopted auto-evaluation method. However, its reliability is compromised by the CoT reasoning's inability to capture comprehensive and deeper details, often leading to incomplete outcomes. Existing methods mainly rely on majority voting or criteria expansion, which is insufficient to address the limitation in CoT. We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses, thereby exposing deeper and more comprehensive details within the candidate responses. This process effectively guides LLM-as-a-Judge to provide a more detailed CoT judgment. Extensive experiments demonstrate that our approach enhances evaluation reliability, achieving an average accuracy gain of 6.7% across five benchmarks. Moreover, our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling for supervised fine-tuning (SFT), referred to as crowd rejection sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs generated by ours are more comprehensive and of higher quality, and evaluation accuracy improves as inference scales.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 48 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

22 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge30 Jan
  2. 2024DeepSeek-V3 Technical Report27 Dec

Other references · 20

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  5. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  7. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  8. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  9. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  10. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  11. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  12. 2024Generative Reward Modelscited by 148
8 more
  1. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  2. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  3. 2024Self-Taught Evaluatorscited by 72
  4. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  5. 2024Direct Judgement Preference Optimizationcited by 33
  6. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  7. 2024Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QAcited by 25
  8. 2024Black-box Uncertainty Quantification Method for LLM-as-a-Judgecited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 11.

19 papers.

Sort:

By area:

  1. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  2. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  3. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 27
  4. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
  5. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  6. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 11
  7. 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Usecited by 10
  8. 2025Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoningcited by 6
  9. 2025SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluationcited by 6
  10. 2026SRR-Judge: Step-Level Rating and Refinement for Enhancing Search-Integrated Reasoning in Search Agentscited by 5
  11. 2025But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectorscited by 3
  12. 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarioscited by 3
7 more
  1. 2026Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translationscited by 2
  2. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  3. 2026Thomson: Continual Learning of Frontier Models for SovereignAIcited by 2
  4. 2026Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Modelscited by 1
  5. 2026Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Modelscited by 1
  6. 2026Auditing Stealth Sycophancy in Mental-Health Dialogue: Structured Clinical-State Diagnostics and Clean Matched Benchmarkscited by 1
  7. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  2. 2025Verdict: A Library for Scaling Judge-Time Compute
  3. 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
  4. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  5. 2025Improving LLM-as-a-Judge Inference with the Judgment Distribution
  6. 2025Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
  7. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  8. 2026JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols