mlsanityData to 14 Oct 2026

EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance

arXiv 2509.23730 ↗First posted 28 Sep 2025cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) have recently advanced in reasoning when optimized with reinforcement learning (RL) under verifiable rewards. Existing methods primarily rely on outcome-based supervision to strengthen internal LLM reasoning, often leading to inefficient exploration and sparse rewards. To mitigate this issue, we propose Expert-Assisted Policy Optimization (EAPO), a novel RL framework that enhances exploration by incorporating multi-turn interactions with external experts during training. Unlike prior methods, where policies reason in isolation, EAPO incentivizes the policy to adaptively determine when and how to consult experts, yielding richer reward signals and more reliable reasoning trajectories. External assistance ultimately internalizes expert knowledge into the policy model, amplifying the model's inherent reasoning capabilities. During evaluation, the policy model has been well-optimized to solve questions independently, producing improved reasoning paths and more accurate solutions. On AIME 2024/2025 and AIMO 2025, EAPO consistently outperforms expert-assisted, expert-distilled, and RL baselines, averaging a 5-point gain over self-exploration RL, and also generalizes to non-math benchmarks, including HumanEval, HLE, GPQA, MMLU, EvalPlus, HotpotQA, and SimpleQA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 46 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

35 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
21 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2018Born Again Neural Networkscited by 546
  3. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  4. 2016The Option-Critic Architecturecited by 448
  5. 2015Policy Distillationcited by 440
  6. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  7. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  8. 2024Measuring short-form factuality in large language modelscited by 373
  9. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  10. 2018Self-Imitation Learningcited by 138
  11. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  12. 2018Kickstarting Deep Reinforcement Learningcited by 81
  13. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  14. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  15. 2020Emergent Social Learning via Multi-agent Reinforcement Learningcited by 44
  16. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 21
  17. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  18. 2025Learning from Peers in Reasoning Modelscited by 16
  19. 2024Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedbackcited by 11
  20. 2024Efficient Active Imitation Learning with Random Network Distillationcited by 7
  21. 2024Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralizationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards
  2. 2026AIPO: Learning to Reason from Active Interaction
  3. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
  4. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  5. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
  6. 2025Agentic Reinforced Policy Optimization
  7. 2025AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin
  8. 2026Experience Augmented Policy Optimization for LLM Reasoning