mlsanityData to 7 Oct 2026

ProCeedRL: Process Critic with Exploratory Demonstration Reinforcement Learning for LLM Agentic Reasoning

arXiv 2604.02006 ↗First posted 2 Apr 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) significantly enhances the reasoning abilities of large language models (LLMs), yet applying it to multi-turn agentic tasks remains challenging due to the long-horizon nature of interactions and the stochasticity of environmental feedback. We identify a structural failure mode in agentic exploration: suboptimal actions elicit noisy observations into misleading contexts, which further weaken subsequent decision-making, making recovery increasingly difficult. This cumulative feedback loop of errors renders standard exploration strategies ineffective and susceptible to the model's reasoning and the environment's randomness. To mitigate this issue, we propose ProCeedRL: Process Critic with Explorative Demonstration RL, shifting exploration from passive selection to active intervention. ProCeedRL employs a process-level critic to monitor interactions in real time, incorporating reflection-based demonstrations to guide agents in stopping the accumulation of errors. We find that this approach significantly exceeds the model's saturated exploration performance, demonstrating substantial exploratory benefits. By learning from exploratory demonstrations and on-policy samples, ProCeedRL significantly improves exploration efficiency and achieves superior performance on complex deep search and embodied tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 41 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

31 of 41 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Agentic Reinforced Policy Optimization26 Jul
  2. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  3. 2025Learning to Reason under Off-Policy Guidance21 Apr
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  8. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2024OpenAI o1 System Cardcited by 1,220
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  7. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  8. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  9. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  10. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  11. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  12. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
10 more
  1. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  2. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  3. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  4. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  5. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  6. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  7. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  8. 2025ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewardscited by 10
  9. 2025RE-Searcher: Robust Agentic Search with Goal-oriented Planning and Self-reflectioncited by 5
  10. 2025MARGE: Improving Math Reasoning for LLMs with Guided Explorationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
  2. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  3. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay
  4. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
  5. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  6. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  7. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey