mlsanityData to 7 Oct 2026

RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Models

arXiv 2603.18859 ↗First posted 19 Mar 2026cs.AI cs.CL cs.LGcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Reinforcement learning (RL) shows promise for enhancing LLM agentic reasoning, yet sparse terminal rewards hinder fine-grained optimization. Process reward modeling offers an alternative but incurs high computational costs, reward hacking risks, and annotation bottlenecks. We introduce RewardFlow, a lightweight method for estimating state-level rewards in agentic reasoning. By constructing state graphs that capture the intrinsic topological structure of trajectories, RewardFlow performs topology-aware propagation to estimate each state's contribution to success, yielding principled, annotation-free dense rewards. Used for RL optimization, RewardFlow substantially outperforms prior baselines across four agentic benchmarks: +6.2% average success rate on text-based tasks, +29.7% on visual reasoning over the strongest baseline across three model scales, and +10% accuracy on DeepResearch, with superior robustness and training efficiency. The implementation of RewardFlow is publicly available at https://github.com/tmlr-group/RewardFlow.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 61 of its 79 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

61 of 79 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  2. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  3. 2025Single-stream Policy Optimization16 Sep
  4. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  5. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models1 Aug
  6. 2025Agentic Reinforced Policy Optimization26 Jul
  7. 2025Learning to Reason without External Rewards26 May
  8. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  9. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  10. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  11. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
8 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025Process Reinforcement through Implicit Rewards3 Feb
  3. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  6. 2023Let's Verify Step by Step31 May
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 41

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  6. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
29 more
  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  2. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  4. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  5. 2018MolGAN: An implicit generative model for small molecular graphscited by 392
  6. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  7. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  8. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  9. 2018Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generationcited by 363
  10. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  11. 2026Reinforcement Learning via Self-Distillationcited by 285
  12. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  13. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  14. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  15. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  16. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  17. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  18. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  19. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  20. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  21. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  22. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  23. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  24. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  25. 2026Reinforcement-aware Knowledge Distillation for LLM Reasoningcited by 29
  26. 2020Induction and Exploitation of Subgoal Automata for Reinforcement Learningcited by 26
  27. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
  28. 2025Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigationcited by 8
  29. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agentscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoningcited by 4
  2. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agentscited by 2
  3. 2026MoFlow: Multi-Objective Agentic Workflow Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Verifiable Process Rewards for Agentic Reasoning
  2. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifier
  3. 2026ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL
  4. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  5. 2026Exploring Reasoning Reward Model for Agents
  6. 2026RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
  7. 2025Demystifying Reinforcement Learning in Agentic Reasoning
  8. 2026Graph Reasoning Paradigm: Structured and Symbolic Reasoning with Topology-Aware Reinforcement Learning for Large Language Models