An Empirical Study of Deep Reinforcement Learning in Continuing Tasks
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
In reinforcement learning (RL), continuing tasks refer to tasks where the agent-environment interaction is ongoing and can not be broken down into episodes. These tasks are suitable when environment resets are unavailable, agent-controlled, or predefined but where all rewards-including those beyond resets-are critical. These scenarios frequently occur in real-world applications and can not be modeled by episodic tasks. While modern deep RL algorithms have been extensively studied and well understood in episodic tasks, their behavior in continuing tasks remains underexplored. To address this gap, we provide an empirical study of several well-known deep RL algorithms using a suite of continuing task testbeds based on Mujoco and Atari environments, highlighting several key insights concerning continuing tasks. Using these testbeds, we also investigate the effectiveness of a method for improving temporal-difference-based RL algorithms in continuing tasks by centering rewards, as introduced by Naik et al. (2024). While their work primarily focused on this method in conjunction with Q-learning, our results extend their findings by demonstrating that this method is effective across a broader range of algorithms, scales to larger tasks, and outperforms two other reward-centering approaches.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 28 references that are in the map. First posted on arXiv 12 Jan 2025.
Builds on
18 of 28 references.
Closest ancestors · 2
Same direction, newest first
Other references · 16
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2016OpenAI Gymcited by 1,736
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
- 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 90
- 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
- 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
- 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 30
- 2020Jelly Bean World: A Testbed for Never-Ending Learningcited by 19
- 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 19
4 more
- 2020Q-learning with Uniformly Bounded Variance: Large Discounting is Not a Barrier to Fast Learningcited by 13
- 2023Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factorcited by 13
- 2023Off-Policy Average Reward Actor-Critic with Deterministic Policy Searchcited by 10
- 2023Pearl: A Production-ready Reinforcement Learning Agentcited by 9
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025A Survey of Continual Reinforcement Learning
- 2025Continual Reinforcement Learning via Autoencoder-Driven Task and New Environment Recognition
- 2019Continual Learning Using World Models for Pseudo-Rehearsal
- 2025The Cell Must Go On: Agar.io for Continual Reinforcement Learning
- 2025Rethinking the Foundations for Continual Reinforcement Learning
- 2025Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines
- 2025MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning
- 2026Position: Deployed Reinforcement Learning should be Continual