# Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling

arXiv: https://arxiv.org/abs/2508.05118 · HTML: https://mlsanity.com/paper/2508.05118/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://mlsanity.com/paper/2409.03215/) (2024)
- [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024)
- [Reinforcement Learning Enhanced LLMs: A Survey](https://mlsanity.com/paper/2412.10400/) (2024)
- [Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning](https://mlsanity.com/paper/2505.00024/) (2025)
- [LLM With Tools: A Survey](https://mlsanity.com/paper/2409.18807/) (2024)
- [e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs](https://mlsanity.com/paper/2506.09026/) (2025)
- [Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach](https://mlsanity.com/paper/2306.03604/) (2023)
- [InSpire: Vision-Language-Action Models with Intrinsic Spatial Reasoning](https://mlsanity.com/paper/2505.13888/) (2025)
- [SFR-RAG: Towards Contextually Faithful LLMs](https://mlsanity.com/paper/2409.09916/) (2024)
- [FunReason: Enhancing Large Language Models' Function Calling via Self-Refinement Multiscale Loss and Automated Data Refinement](https://mlsanity.com/paper/2505.20192/) (2025)
- [Policy Newton Algorithm in Reproducing Kernel Hilbert Space](https://mlsanity.com/paper/2506.01597/) (2025)

## Built on by (most cited first, 7 of 7)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL](https://mlsanity.com/paper/2605.18703/) (2026)
- [FunReason-MT Technical Report: Advanced Data Synthesis Solution for Real-world Multi-Turn Tool-use](https://mlsanity.com/paper/2510.24645/) (2025)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [From Failure to Mastery: Generating Hard Samples for Tool-use Agents](https://mlsanity.com/paper/2601.01498/) (2026)
- [Leveraging Error Diversity in Group Rollouts for Reinforcement Learning](https://mlsanity.com/paper/2605.17333/) (2026)
- [What Is Preference Optimization Doing, and Why?](https://mlsanity.com/paper/2512.00778/) (2025)

## Abstract (arXiv)

The effective training of Large Language Models (LLMs) for function calling faces a critical challenge: balancing exploration of complex reasoning paths with stable policy optimization. Standard methods like Supervised Fine-Tuning (SFT) fail to instill robust reasoning, and traditional Reinforcement Learning (RL) struggles with inefficient exploration. We propose EGPO, a new RL framework built upon Group Relative Policy Optimization (GRPO), designed to address this challenge directly. The core of EGPO is an entropy-enhanced advantage function that integrates the entropy of the model's Chain-of-Thought (CoT) into the policy gradient computation. This encourages the generation of diverse reasoning strategies. To maintain optimization direction, the entropy bonus is carefully constrained by a clipping mechanism. Complemented by a strict, binary reward signal, EGPO effectively guides the model towards discovering structured and accurate tool invocation patterns. On the challenging Berkeley Function Calling Leaderboard (BFCL), a 4B-parameter model trained with EGPO sets a new state-of-the-art among models of comparable size, surpassing a range of strong competitors, including GPT-4o and Gemini-2.5.