mlsanityData to 7 Oct 2026

MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation

arXiv 2502.12468 ↗First posted 18 Feb 2025cs.LG cs.AIcited by 32

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

The LLM-as-a-Judge paradigm shows promise for evaluating generative content but lacks reliability in reasoning-intensive scenarios, such as programming. Inspired by recent advances in reasoning models and shifts in scaling laws, we pioneer bringing test-time computation into LLM-as-a-Judge, proposing MCTS-Judge, a resource-efficient, System-2 thinking framework for code correctness evaluation. MCTS-Judge leverages Monte Carlo Tree Search (MCTS) to decompose problems into simpler, multi-perspective evaluations. Through a node-selection strategy that combines self-assessment based on historical actions in the current trajectory and the Upper Confidence Bound for Trees based on prior rollouts, MCTS-Judge balances global optimization and refinement of the current trajectory. We further designed a high-precision, unit-test-level reward mechanism to encourage the Large Language Model (LLM) to perform line-by-line analysis. Extensive experiments on three benchmarks and five LLMs demonstrate the effectiveness of MCTS-Judge, which improves the base model's accuracy from 41% to 80%, surpassing the o1-series models with 3x fewer tokens. Further evaluations validate the superiority of its reasoning trajectory in logic, analytics, thoroughness, and overall quality, while revealing the test-time scaling law of the LLM-as-a-Judge paradigm.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 38 of its 55 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

38 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Dynamic Scaling of Unit Tests for Code Reward Modeling2 Jan
  2. 2024OpenAI o1 System Card21 Dec
  3. 2024Qwen2.5-Coder Technical Report18 Sep
  4. 2023Reasoning with Language Model is Planning with World Model24 May

Other references · 34

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2024A Survey on LLM-as-a-Judgecited by 1,384
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  9. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  11. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  12. 2022CodeT: Code Generation with Generated Testscited by 369
22 more
  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  3. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  4. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  5. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  6. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  7. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  8. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  9. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  10. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  11. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  12. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  13. 2024MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notescited by 77
  14. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  15. 2024MarsCode Agent: AI-native Automated Bug Fixingcited by 68
  16. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  17. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 41
  18. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Timecited by 40
  19. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  20. 2024Evaluating and Aligning CodeLLMs on Human Preferencecited by 22
  21. 2024Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitationcited by 19
  22. 2025On Iterative Evaluation and Enhancement of Code Quality Using GPT-4ocited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 15.

32 papers.

Sort:

By area:

  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  2. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  3. 2025Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Modelscited by 30
  4. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  5. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  6. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 10
  7. 2025First Finish Search: Efficient Test-Time Scaling in Large Language Modelscited by 8
  8. 2025VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Codecited by 7
  9. 2025Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specificationscited by 5
  10. 2025MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Explorationcited by 4
  11. 2025Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planningcited by 4
  12. 2025CALM: A Framework for Continuous, Adaptive, and LLM-Mediated Anomaly Detection in Time-Series Streamscited by 4
20 more
  1. 2025Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scalingcited by 4
  2. 2026TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RLcited by 4
  3. 2026Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgementcited by 4
  4. 2026OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimizationcited by 3
  5. 2025Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Modelscited by 2
  6. 2025Becoming Experienced Judges: Selective Test-Time Learning for Evaluatorscited by 2
  7. 2025Multicalibration for LLM-based Code Generationcited by 2
  8. 2026GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Modelscited by 2
  9. 2025CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfercited by 1
  10. 2026Simple Role Assignment is Extraordinarily Effective for Safety Alignmentcited by 1
  11. 2026Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Modelscited by 1
  12. 2026VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selectioncited by 1
  13. 2026Intrinsic Selection and Particle Resampling for Inference-Time Scaling Beyond Domain Verifiabilitycited by 1
  14. 2026Interpretable Adaptive Sampling for LLM Test-Time Scalingcited by 1
  15. 2026Self-Evolving Multi-Agent Framework for Efficient Decision Making in Real-Time Strategy Scenarioscited by 0
  16. 2026Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safetycited by 0
  17. 2026Entropy Centroids as Intrinsic Rewards for Test-Time Scalingcited by 0
  18. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0
  19. 2026RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generatorcited by 0
  20. 2026Geometry-Aware MCTS for Extremal Problems in Combinatorial Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
  2. 2025S*: Test Time Scaling for Code Generation
  3. 2026DAJ: Data-Reweighted LLM Judge for Test-Time Scaling in Code Generation
  4. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
  5. 2026CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation
  6. 2026ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning
  7. 2026LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB
  8. 2025Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey