# RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use

arXiv: https://arxiv.org/abs/2509.06980 · HTML: https://mlsanity.com/paper/2509.06980/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 5 of its 6 references that are in the map.
- First posted on arXiv 31 Aug 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 5 of 5)

- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.01441/) (2025)
- [Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving](https://mlsanity.com/paper/2505.07773/) (2025)
- [Robust Planning with LLM-Modulo Framework: Case Study in Travel Planning](https://mlsanity.com/paper/2405.20625/) (2024)
- [TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning](https://mlsanity.com/paper/2502.20508/) (2025)

## Built on by (most cited first, 20 of 20)

- [From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory](https://mlsanity.com/paper/2511.07800/) (2025)
- [Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning](https://mlsanity.com/paper/2602.00994/) (2026)
- [TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents](https://mlsanity.com/paper/2607.13988/) (2026)
- [MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQL](https://mlsanity.com/paper/2510.25510/) (2025)
- [ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIs](https://mlsanity.com/paper/2512.16149/) (2025)
- [LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services](https://mlsanity.com/paper/2512.07436/) (2025)
- [Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Use](https://mlsanity.com/paper/2602.11541/) (2026)
- [SeeUPO: Sequence-Level Agentic-RL with Convergence Guarantees](https://mlsanity.com/paper/2602.06554/) (2026)
- [SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents](https://mlsanity.com/paper/2604.07791/) (2026)
- [AlphaApollo: A System for Deep Agentic Reasoning](https://mlsanity.com/paper/2510.06261/) (2025)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness](https://mlsanity.com/paper/2509.23206/) (2025)
- [Training Multi-Image Vision Agents via End2End Reinforcement Learning](https://mlsanity.com/paper/2512.08980/) (2025)
- [Agentic Tool Use in Large Language Models](https://mlsanity.com/paper/2604.00835/) (2026)
- [CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?](https://mlsanity.com/paper/2610.07557/) (2026)
- [Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning](https://mlsanity.com/paper/2602.21517/) (2026)
- [From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation](https://mlsanity.com/paper/2607.19395/) (2026)
- [Mint-Agent: Introducing Finance-Native Agentic Foundation Models](https://mlsanity.com/paper/2608.16386/) (2026)
- [MCPDisCo: Seesaw-Mitigated Tuning for MCP-Native Agents](https://mlsanity.com/paper/2610.14956/) (2026)
- [WorkRover: Scaling Long-Horizon Cowork Agent Training Via Recursive Environment Evolution](https://mlsanity.com/paper/2610.18067/) (2026)

## Abstract (arXiv)

Large language models excel at basic reasoning but struggle with tasks that require interaction with external tools. We present RLFactory, a plug-and-play reinforcement learning post-training framework for multi-round tool use. RLFactory tackles (i) tool-call stability and adaptability amid tool heterogeneity and interface issues via an asyncio-based asynchronous caller and a decoupled tool/training architecture, and (ii) diverse evaluation needs via a reward layer supporting rule-based, model-judgment, and tool-verification signals. It reconstructs the MDP by introducing observation markers from tool feedback, closing the loop among model, tools, and environment, and implements a generate-parse-invoke-update workflow for dynamic policy optimization. On Search-R1 with Qwen3-4B, RLFactory achieves a 0.486 test score on the Natural Questions (NQ) dataset, surpassing larger models trained with similar techniques (e.g., Qwen2.5-7B-Instruct-GRPO at 0.473), and increases training throughput by 6.8x. RLFactory provides a low-barrier, highly adaptable framework for strengthening multi-round tool use of LLMs in real-world scenarios. Code: https://github.com/Simple-Efficient/RL-Factory.