# Reducing Cognitive Overhead in Tool Use via Multi-Small-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2508.08882 · HTML: https://mlsanity.com/paper/2508.08882/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Process Reinforcement through Implicit Rewards](https://mlsanity.com/paper/2502.01456/) (2025)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2310.03731/) (2023)
- [OpenAgents: An Open Platform for Language Agents in the Wild](https://mlsanity.com/paper/2310.10634/) (2023)
- [LLM Multi-Agent Systems: Challenges and Open Problems](https://mlsanity.com/paper/2402.03578/) (2024)
- [G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks](https://mlsanity.com/paper/2410.11782/) (2024)
- [Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools](https://mlsanity.com/paper/2502.04644/) (2025)
- [Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback](https://mlsanity.com/paper/2506.03106/) (2025)
- [Language Agents as Optimizable Graphs](https://mlsanity.com/paper/2402.16823/) (2024)
- [MALT: Improving Reasoning with Multi-Agent LLM Training](https://mlsanity.com/paper/2412.01928/) (2024)
- [MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems](https://mlsanity.com/paper/2404.04735/) (2024)
- [ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Tasks](https://mlsanity.com/paper/2503.02390/) (2025)
- [Reinforce LLM Reasoning through Multi-Agent Reflection](https://mlsanity.com/paper/2506.08379/) (2025)

## Built on by (most cited first, 3 of 3)

- [LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review](https://mlsanity.com/paper/2604.16321/) (2026)
- [Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Long-Horizon Workflows](https://mlsanity.com/paper/2604.22820/) (2026)
- [Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act](https://mlsanity.com/paper/2609.16268/) (2026)

## Abstract (arXiv)

Recent advances in multi-agent systems highlight the potential of specialized small agents that collaborate via division of labor. Existing tool-integrated reasoning systems, however, often follow a single-agent paradigm in which one large model interleaves long-horizon reasoning with precise tool operations, leading to cognitive-load interference and unstable coordination. We present MSARL, a Multi-Small-Agent Reinforcement Learning framework that explicitly decouples reasoning from tool use. In MSARL, a Reasoning Agent decomposes problems and plans tool invocations, while multiple Tool Agents specialize in specific external tools, each trained via a combination of imitation learning and reinforcement learning with role-specific rewards. On mathematical problem solving with code execution, MSARL significantly improves reasoning stability and final-answer accuracy over single-agent baselines. Moreover, the architecture generalizes to diverse tool-use tasks, demonstrating that cognitive-role decoupling with small agents is a scalable blueprint for multi-agent AI design.