mlsanityData to 14 Oct 2026

Ghost Policies: A New Paradigm for Understanding and Learning from Failure in Deep Reinforcement Learning

arXiv 2506.12366 ↗First posted 14 Jun 2025cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Deep Reinforcement Learning (DRL) agents often exhibit intricate failure modes that are difficult to understand, debug, and learn from. This opacity hinders their reliable deployment in real-world applications. To address this critical gap, we introduce ``Ghost Policies,'' a concept materialized through Arvolution, a novel Augmented Reality (AR) framework. Arvolution renders an agent's historical failed policy trajectories as semi-transparent ``ghosts'' that coexist spatially and temporally with the active agent, enabling an intuitive visualization of policy divergence. Arvolution uniquely integrates: (1) AR visualization of ghost policies, (2) a behavioural taxonomy of DRL maladaptation, (3) a protocol for systematic human disruption to scientifically study failure, and (4) a dual-learning loop where both humans and agents learn from these visualized failures. We propose a paradigm shift, transforming DRL agent failures from opaque, costly errors into invaluable, actionable learning resources, laying the groundwork for a new research field: ``Failure Visualization Learning.''

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 7 of its 17 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

7 of 17 references.

References · 7

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  3. 2019Solving Rubik's Cube with a Robot Handcited by 734
  4. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  5. 2018Unity: A General Platform for Intelligent Agentscited by 236
  6. 2023Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomescited by 14
  7. 2024Toward Debugging Deep Reinforcement Learning Programs with RLExplorercited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning for Large Model: A Survey
  2. 2024Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models
  3. 2026Learning from the Test: Self-Referential Differential Testing for Deep RL Agents
  4. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress
  5. 2025An Introduction to Deep Reinforcement and Imitation Learning
  6. 2020Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning
  7. 2026Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models
  8. 2026Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models