mlsanityData to 13 Oct 2026

Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

arXiv 2510.23038 ↗First posted 27 Oct 2025cs.CL cs.AI cs.LGcited by 16

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 80 of its 86 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

80 of 86 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use1 Sep
  2. 2025Understanding Tool-Integrated Reasoning26 Aug
  3. 2025WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning28 May
  4. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning22 May
  5. 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently21 Apr
  6. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  7. 2025Dynamic Scaling of Unit Tests for Code Reward Modeling2 Jan

Other references · 73

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  5. 2024GPT-4o System Cardcited by 4,536
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,714
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,729
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,968
  9. 2025Gemma 3 Technical Reportcited by 1,748
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,399
  11. 2025s1: Simple test-time scalingcited by 1,322
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
38 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,066
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,020
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 913
  4. 2024Self-Rewarding Language Modelscited by 666
  5. 2022Large Language Models Can Self-Improvecited by 659
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 648
  7. 2022PAL: Program-aided Language Modelscited by 647
  8. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 542
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 533
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 488
  11. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 470
  12. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 371
  13. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 337
  14. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 329
  15. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 324
  16. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 304
  17. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 247
  18. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 236
  19. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 231
  20. 2023Proving Test Set Contamination in Black Box Language Modelscited by 228
  21. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  22. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 172
  23. 2025ToRL: Scaling Tool-Integrated RLcited by 168
  24. 2023Generative Judge for Evaluating Alignmentcited by 167
  25. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  26. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  27. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 141
  28. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 134
  29. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 125
  30. 2025Process Reward Models That Thinkcited by 118
  31. 2024Critique-out-Loud Reward Modelscited by 105
  32. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 95
  33. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 93
  34. 2024How to Evaluate Reward Models for RLHFcited by 84
  35. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 77
  36. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  37. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 74
  38. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 50
  2. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 29
  3. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 17
  4. 2026Agent-as-a-Judgecited by 16
  5. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifiercited by 14
  6. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluationcited by 11
  7. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 6
  8. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 4
  9. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 4
  10. 2026Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learningcited by 3
  11. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  12. 2026Evaluating Agentic Configuration Repair for Computer Networkscited by 2
4 more
  1. 2026IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Modelscited by 1
  2. 2026CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteriacited by 1
  3. 2026Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learningcited by 1
  4. 2026WMLLM: Self-Evolving Optimization Agents via Predict-Then-Act World Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
  2. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
  3. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
  4. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
  5. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
  6. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
  7. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  8. 2025JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation