mlsanityData to 14 Oct 2026

Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning

arXiv 2505.00024 ↗First posted 25 Apr 2025cs.CL cs.AIcited by 68

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 34 of its 52 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

34 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  2. 2023ART: Automatic multi-step reasoning and tool-use for large language models16 Mar

Other references · 32

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  11. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  12. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
20 more
  1. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 303
  2. 2021Internet-Augmented Dialogue Generationcited by 217
  3. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
  4. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  5. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  6. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  7. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  8. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  9. 2024Process Reward Model with Q-Value Rankingscited by 80
  10. 2025ACEBench: Who Wins the Match Point in Tool Usage?cited by 71
  11. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 68
  12. 2025SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learningcited by 68
  13. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 54
  14. 2025MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaversecited by 51
  15. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 39
  16. 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
  17. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learningcited by 25
  18. 2024LATTE: Learning to Think with Vision Specialistscited by 14
  19. 2024EcoAct: Economic Agent Determines When to Register What Actioncited by 11
  20. 2025iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Usecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 42.

68 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  3. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 79
  4. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 63
  5. 2025Towards General Agentic Intelligence via Environment Scalingcited by 54
  6. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 47
  7. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  8. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  9. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
  10. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 30
  11. 2026Flow caching for autoregressive video generationcited by 25
  12. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
38 more
  1. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 21
  2. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 20
  3. 2025Reasoning-Table: Exploring Reinforcement Learning for Table Reasoningcited by 18
  4. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 18
  5. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 17
  6. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  7. 2025StepFun-Prover Preview: Let's Think and Verify Step by Stepcited by 13
  8. 2026α3-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networkscited by 10
  9. 2026MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matchingcited by 10
  10. 2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
  11. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  12. 2026From Failure to Mastery: Generating Hard Samples for Tool-use Agentscited by 8
  13. 2026Workflow-R1: Group Sub-sequence Policy Optimization for Multi-turn Workflow Constructioncited by 8
  14. 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Modelscited by 8
  15. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  16. 2025Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Callingcited by 7
  17. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix Itcited by 7
  18. 2026Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learningcited by 6
  19. 2025FunReason: Enhancing Large Language Models' Function Calling via Self-Refinement Multiscale Loss and Automated Data Refinementcited by 5
  20. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 5
  21. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generationcited by 5
  22. 2025PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awarenesscited by 4
  23. 2025ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMscited by 4
  24. 2025PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoningcited by 4
  25. 2025CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questionscited by 4
  26. 2026RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinementcited by 4
  27. 2025How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on τ-benchcited by 3
  28. 2025Natural Language Tools: A Natural Language Approach to Tool Calling In Large Language Agentscited by 3
  29. 2026Video-Based Reward Modeling for Computer-Use Agentscited by 3
  30. 2026LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Modelscited by 3
  31. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 3
  32. 2026Pushing the Limits of LLM Tool Calling via Experiential Knowledge Integration and Activationcited by 3
  33. 2026TCPO: Turn-Level Credit Policy Optimizationcited by 3
  34. 2026PACE: Prefix-Protected and Difficulty-Aware Compression for Efficient Reasoningcited by 2
  35. 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Trainingcited by 2
  36. 2026Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Callingcited by 2
  37. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debatecited by 1
  38. 2026Try, Check and Retry: A Divide-and-Conquer Framework for Boosting Long-context Tool-Calling Performance of LLMscited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch
  2. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
  3. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
  4. 2026Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees
  5. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  6. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
  7. 2026Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning
  8. 2025Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning