Data to 14 Oct 2026

Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge

arXiv 2501.18099 ↗First posted 30 Jan 2025cs.AI cs.CLcited by 78

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

LLM-as-a-Judge models generate chain-of-thought (CoT) sequences intended to capture the step-bystep reasoning process that underlies the final evaluation of a response. However, due to the lack of human annotated CoTs for evaluation, the required components and structure of effective reasoning traces remain understudied. Consequently, previous approaches often (1) constrain reasoning traces to hand-designed components, such as a list of criteria, reference answers, or verification questions and (2) structure them such that planning is intertwined with the reasoning for evaluation. In this work, we propose EvalPlanner, a preference optimization algorithm for Thinking-LLM-as-a-Judge that first generates an unconstrained evaluation plan, followed by its execution, and then the final judgment. In a self-training loop, EvalPlanner iteratively optimizes over synthetically constructed evaluation plans and executions, leading to better final verdicts. Our method achieves a new state-of-the-art performance for generative reward models on RewardBench (with a score of 93.9), despite being trained on fewer amount of, and synthetically generated, preference pairs. Additional experiments on other benchmarks like RM-Bench, JudgeBench, and FollowBenchEval further highlight the utility of both planning and reasoning for building robust LLM-as-a-Judge reasoning models.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 52 of its 56 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

52 of 56 references.

Closest ancestors · 1

Same direction, newest first

2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 51

2022Training language models to follow instructions with human feedbackcited by 15,971
2024The Llama 3 Herd of Modelscited by 14,715
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2023Let's Verify Step by Stepcited by 3,789
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
2021Ethical and social risks of harm from Language Modelscited by 966
2022STaR: Bootstrapping Reasoning With Reasoningcited by 940

38 more

2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
2024Self-Rewarding Language Modelscited by 681
2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
2024Iterative Reasoning Preference Optimizationcited by 246
2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
2023Self-Alignment with Instruction Backtranslationcited by 151
2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
2024Critique-out-Loud Reward Modelscited by 106
2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
2024How to Evaluate Reward Models for RLHFcited by 88
2024Self-Taught Evaluatorscited by 72
2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 70
2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
2024Improving Reward Models with Synthetic Critiquescited by 44
2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
2024Direct Judgement Preference Optimizationcited by 33
2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
2024Self-rationalization improves LLM as a fine-grained judgecited by 27

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 44; 2026: 34.

78 papers.

Sort:

By area:

2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 24

38 more

2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 21
2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 20
2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 19
2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
2025Theorem Prover as a Judge for Synthetic Data Generationcited by 14
2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 13
2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 11
2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 11
2026Reward Modeling from Natural Language Human Feedbackcited by 11
2025A Survey of Scaling in Large Language Model Reasoningcited by 10
2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probescited by 9
2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
2025LLMs for Customized Marketing Content Generation and Evaluation at Scalecited by 9
2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 8
2025Efficient LLM Collaboration via Planningcited by 7
2025CML-Bench: A Framework for Evaluating and Enhancing LLM-Powered Movie Scripts Generationcited by 7
2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
2025Self-Improving VLM Judges Without Human Annotationscited by 7
2025Expert Preference-based Evaluation of Automated Related Work Generationcited by 6
2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 5
2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
2025TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevancecited by 4
2026Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steeringcited by 4
2025Judging with Confidence: Calibrating Autoraters to Preference Distributionscited by 3
2025CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Researchcited by 3
2026Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenescited by 3
2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
2026MJ1: Multimodal Judgment via Grounded Verificationcited by 3
2026Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modelingcited by 3
2025Interpreting LLM-as-a-Judge Policies via Verifiable Global Explanationscited by 2
2025Budget-aware Test-time Scaling via Discriminative Verificationcited by 2
2025Becoming Experienced Judges: Selective Test-Time Learning for Evaluatorscited by 2

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way