# RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents

arXiv: https://arxiv.org/abs/2507.22844 · HTML: https://mlsanity.com/paper/2507.22844/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2025.

## Built on by (most cited first, 30 of 58)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Hindsight Credit Assignment for Long-Horizon LLM Agents](https://mlsanity.com/paper/2603.08754/) (2026)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Agentic Reinforcement Learning with Implicit Step Rewards](https://mlsanity.com/paper/2509.19199/) (2025)
- [SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph](https://mlsanity.com/paper/2510.20022/) (2025)
- [EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection](https://mlsanity.com/paper/2603.04900/) (2026)
- [DeepWideSearch: Benchmarking Depth and Width in Agentic Information Seeking](https://mlsanity.com/paper/2510.20168/) (2025)
- [Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization](https://mlsanity.com/paper/2602.23008/) (2026)
- [Milestone-Guided Policy Learning for Long-Horizon Language Agents](https://mlsanity.com/paper/2605.06078/) (2026)
- [EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning](https://mlsanity.com/paper/2509.22576/) (2025)
- [Exploring Reasoning Reward Model for Agents](https://mlsanity.com/paper/2601.22154/) (2026)
- [Stop-RAG: Value-Based Retrieval Control for Iterative RAG](https://mlsanity.com/paper/2510.14337/) (2025)
- [PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost](https://mlsanity.com/paper/2603.21383/) (2026)
- [Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning](https://mlsanity.com/paper/2509.22601/) (2025)
- [Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning](https://mlsanity.com/paper/2601.20209/) (2026)
- [Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration](https://mlsanity.com/paper/2602.03647/) (2026)
- [Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty](https://mlsanity.com/paper/2603.15500/) (2026)
- [WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning](https://mlsanity.com/paper/2601.03164/) (2026)
- [RoboAgent: Chaining Basic Capabilities for Embodied Task Planning](https://mlsanity.com/paper/2604.07774/) (2026)
- [Recursive Self-Evolving Agents via Held-Out Selection](https://mlsanity.com/paper/2606.28374/) (2026)
- [On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents](https://mlsanity.com/paper/2603.12109/) (2026)
- [TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoning](https://mlsanity.com/paper/2603.25419/) (2026)
- [The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agent from Planning with Actions to Planning with Schemas](https://mlsanity.com/paper/2510.07091/) (2025)
- [Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents](https://mlsanity.com/paper/2602.02050/) (2026)
- [Think Fast and Slow: Step-Level Cognitive Depth Adaptation for LLM Agents](https://mlsanity.com/paper/2602.12662/) (2026)
- [Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security](https://mlsanity.com/paper/2605.23989/) (2026)
- [Bridging the Agent-World Gap: Text World Models for LLM-based Agents](https://mlsanity.com/paper/2606.09032/) (2026)
- [Reducing Belief Deviation in Reinforcement Learning for Active Reasoning](https://mlsanity.com/paper/2510.12264/) (2025)
- [Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learning](https://mlsanity.com/paper/2509.23870/) (2025)
- [Verifying Meta-Awareness via Predictive Rewards in Reasoning Models](https://mlsanity.com/paper/2510.03259/) (2025)

## Abstract (arXiv)

The development of autonomous agents for complex, long-horizon tasks is a central goal in AI. However, dominant training paradigms face a critical limitation: reinforcement learning (RL) methods that optimize solely for final task success often reinforce flawed or inefficient reasoning paths, a problem we term inefficient exploration. This leads to agents that are brittle and fail to generalize, as they learn to find solutions without learning how to reason coherently. To address this, we introduce RLVMR, a novel framework that integrates dense, process-level supervision into end-to-end RL by rewarding verifiable, meta-reasoning behaviors. RLVMR equips an agent to explicitly tag its cognitive steps, such as planning, exploration, and reflection, and provides programmatic, rule-based rewards for actions that contribute to effective problem-solving. These process-centric rewards are combined with the final outcome signal and optimized using a critic-free policy gradient method. On the challenging ALFWorld and ScienceWorld benchmarks, RLVMR achieves new state-of-the-art results, with our 7B model reaching an 83.6% success rate on the most difficult unseen task split. Our analysis confirms these gains stem from improved reasoning quality, including significant reductions in redundant actions and enhanced error recovery, leading to more robust, efficient, and interpretable agents.