Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Deep reinforcement learning (DRL) agents excel in solving complex decision-making tasks across various domains. However, they often require a substantial number of training steps and a vast experience replay buffer, leading to significant computational and resource demands. To address these challenges, we introduce a novel theoretical result that leverages the Neyman-Rubin potential outcomes framework into DRL. Unlike most methods that focus on bounding the counterfactual loss, we establish a causal bound on the factual loss, which is analogous to the on-policy loss in DRL. This bound is computed by storing past value network outputs in the experience replay buffer, effectively utilizing data that is usually discarded. Extensive experiments across the Atari 2600 and MuJoCo domains on various agents, such as DQN and SAC, achieve up to 383% higher reward ratio, outperforming the same agents without our proposed term, and reducing the experience replay buffer size by up to 96%, significantly improving sample efficiency at a negligible cost.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 15 Jul 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning
- 2026WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
- 2024CIER: A Novel Experience Replay Approach with Causal Inference in Deep Reinforcement Learning
- 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning
- 2026Learning from the Near Future: Temporal Self-Distillation for RLVR
- 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning
- 2025Confounding Robust Deep Reinforcement Learning: A Causal Approach
- 2019An Optimistic Perspective on Offline Reinforcement Learning