# Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning

arXiv: https://arxiv.org/abs/2505.00024 · HTML: https://mlsanity.com/paper/2505.00024/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 30 of 34)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning](https://mlsanity.com/paper/2503.07365/) (2025)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023)
- [UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning](https://mlsanity.com/paper/2503.21620/) (2025)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [Internet-augmented language models through few-shot prompting for open-domain question answering](https://mlsanity.com/paper/2203.05115/) (2022)
- [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024)
- [Process Reward Model with Q-Value Rankings](https://mlsanity.com/paper/2410.11287/) (2024)
- [ACEBench: Who Wins the Match Point in Tool Usage?](https://mlsanity.com/paper/2501.12851/) (2025)
- [MathChat: Converse to Tackle Challenging Math Problems with LLM Agents](https://mlsanity.com/paper/2306.01337/) (2023)
- [SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learning](https://mlsanity.com/paper/2504.08600/) (2025)
- [What Are Tools Anyway? A Survey from the Language Model Perspective](https://mlsanity.com/paper/2403.15452/) (2024)
- [MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse](https://mlsanity.com/paper/2503.18470/) (2025)
- [Adaptive In-conversation Team Building for Language Model Agents](https://mlsanity.com/paper/2405.19425/) (2024)
- [Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation](https://mlsanity.com/paper/2503.07826/) (2025)

## Built on by (most cited first, 30 of 68)

- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting](https://mlsanity.com/paper/2508.11408/) (2025)
- [Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment](https://mlsanity.com/paper/2505.11821/) (2025)
- [EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis](https://mlsanity.com/paper/2601.05808/) (2026)
- [Towards General Agentic Intelligence via Environment Scaling](https://mlsanity.com/paper/2509.13311/) (2025)
- [ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration](https://mlsanity.com/paper/2511.21689/) (2025)
- [LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model](https://mlsanity.com/paper/2509.00676/) (2025)
- [Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges](https://mlsanity.com/paper/2604.13602/) (2026)
- [Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning](https://mlsanity.com/paper/2509.06948/) (2025)
- [CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic](https://mlsanity.com/paper/2511.12159/) (2025)
- [Flow caching for autoregressive video generation](https://mlsanity.com/paper/2602.10825/) (2026)
- [Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills](https://mlsanity.com/paper/2512.16301/) (2025)
- [SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL](https://mlsanity.com/paper/2512.04069/) (2025)
- [ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents](https://mlsanity.com/paper/2603.18815/) (2026)
- [Reasoning-Table: Exploring Reinforcement Learning for Table Reasoning](https://mlsanity.com/paper/2506.01710/) (2025)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls](https://mlsanity.com/paper/2511.09148/) (2025)
- [AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent](https://mlsanity.com/paper/2512.20745/) (2025)
- [StepFun-Prover Preview: Let's Think and Verify Step by Step](https://mlsanity.com/paper/2507.20199/) (2025)
- [α3-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networks](https://mlsanity.com/paper/2601.03281/) (2026)
- [MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching](https://mlsanity.com/paper/2601.10712/) (2026)
- [Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models](https://mlsanity.com/paper/2602.02244/) (2026)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [From Failure to Mastery: Generating Hard Samples for Tool-use Agents](https://mlsanity.com/paper/2601.01498/) (2026)
- [Workflow-R1: Group Sub-sequence Policy Optimization for Multi-turn Workflow Construction](https://mlsanity.com/paper/2602.01202/) (2026)
- [On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2602.03392/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling](https://mlsanity.com/paper/2508.05118/) (2025)
- [Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It](https://mlsanity.com/paper/2606.26027/) (2026)
- [Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning](https://mlsanity.com/paper/2604.09813/) (2026)

## Abstract (arXiv)

Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.