# Continual Reinforcement Learning in 3D Non-stationary Environments

arXiv: https://arxiv.org/abs/1905.10112 · HTML: https://mlsanity.com/paper/1905.10112/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 22 of 22)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [Continual World: A Robotic Benchmark For Continual Reinforcement Learning](https://mlsanity.com/paper/2105.10919/) (2021)
- [CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents](https://mlsanity.com/paper/2110.10067/) (2021)
- [Continuous Coordination As a Realistic Scenario for Lifelong Learning](https://mlsanity.com/paper/2103.03216/) (2021)
- [You Only Live Once: Single-Life Reinforcement Learning](https://mlsanity.com/paper/2210.08863/) (2022)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation](https://mlsanity.com/paper/2109.00525/) (2021)
- [Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization](https://mlsanity.com/paper/2209.00347/) (2022)
- [Self-Activating Neural Ensembles for Continual Reinforcement Learning](https://mlsanity.com/paper/2301.00141/) (2022)
- [Lifetime policy reuse and the importance of task capacity](https://mlsanity.com/paper/2106.01741/) (2021)
- [Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning](https://mlsanity.com/paper/2406.06037/) (2024)
- [From Sparse to Dense: Toddler-inspired Reward Transition in Goal-Oriented Reinforcement Learning](https://mlsanity.com/paper/2501.17842/) (2025)
- [Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments](https://mlsanity.com/paper/2109.07855/) (2021)
- [Resilient robot teams: a review integrating decentralised control, change-detection, and learning](https://mlsanity.com/paper/2204.10063/) (2022)
- [Smaller Is Better: An Analysis of Instance Quantity/Quality Trade-off in Rehearsal-based Continual Learning](https://mlsanity.com/paper/2105.14106/) (2021)
- [Architect, Regularize and Replay (ARR): a Flexible Hybrid Approach for Continual Learning](https://mlsanity.com/paper/2301.02464/) (2023)
- [Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spaces](https://mlsanity.com/paper/2506.05702/) (2025)
- [Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot](https://mlsanity.com/paper/2509.19525/) (2025)
- [GEM-Style Constraints for PEFT with Dual Gradient Projection in LoRA](https://mlsanity.com/paper/2601.02500/) (2026)
- [CRoSS: A Continual Robotic Simulation Suite for Scalable Reinforcement Learning with High Task Diversity and Realistic Physics Simulation](https://mlsanity.com/paper/2602.04868/) (2026)

## Abstract (arXiv)

High-dimensional always-changing environments constitute a hard challenge for current reinforcement learning techniques. Artificial agents, nowadays, are often trained off-line in very static and controlled conditions in simulation such that training observations can be thought as sampled i.i.d. from the entire observations space. However, in real world settings, the environment is often non-stationary and subject to unpredictable, frequent changes. In this paper we propose and openly release CRLMaze, a new benchmark for learning continually through reinforcement in a complex 3D non-stationary task based on ViZDoom and subject to several environmental changes. Then, we introduce an end-to-end model-free continual reinforcement learning strategy showing competitive results with respect to four different baselines and not requiring any access to additional supervised signals, previously encountered environmental conditions or observations.