# CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2110.10067 · HTML: https://mlsanity.com/paper/2110.10067/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 90 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 30 of 64)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)

## Built on by (most cited first, 30 of 38)

- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data](https://mlsanity.com/paper/2412.07762/) (2024)
- [Building a Subspace of Policies for Scalable Continual Learning](https://mlsanity.com/paper/2211.10445/) (2022)
- [Lifelong Reinforcement Learning with Modulating Masks](https://mlsanity.com/paper/2212.11110/) (2022)
- [Online Continual Learning For Interactive Instruction Following Agents](https://mlsanity.com/paper/2403.07548/) (2024)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [The Effectiveness of World Models for Continual Reinforcement Learning](https://mlsanity.com/paper/2211.15944/) (2022)
- [Lifelong Learning Metrics](https://mlsanity.com/paper/2201.08278/) (2022)
- [A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems](https://mlsanity.com/paper/2301.07799/) (2023)
- [Dynamic Dialogue Policy for Continual Reinforcement Learning](https://mlsanity.com/paper/2204.05928/) (2022)
- [Beyond Supervised Continual Learning: a Review](https://mlsanity.com/paper/2208.14307/) (2022)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.14990/) (2025)
- [Continual Vision-based Reinforcement Learning with Group Symmetries](https://mlsanity.com/paper/2210.12301/) (2022)
- [Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning](https://mlsanity.com/paper/2207.11584/) (2022)
- [Self-Activating Neural Ensembles for Continual Reinforcement Learning](https://mlsanity.com/paper/2301.00141/) (2022)
- [L2Explorer: A Lifelong Reinforcement Learning Assessment Environment](https://mlsanity.com/paper/2203.07454/) (2022)
- [Evaluating Continual Learning on a Home Robot](https://mlsanity.com/paper/2306.02413/) (2023)
- [A Study of Continual Learning Methods for Q-Learning](https://mlsanity.com/paper/2206.03934/) (2022)
- [System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games](https://mlsanity.com/paper/2212.04603/) (2022)
- [Continual Visual Reinforcement Learning with A Life-Long World Model](https://mlsanity.com/paper/2303.06572/) (2023)
- [Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning](https://mlsanity.com/paper/2412.14865/) (2024)
- [The Cell Must Go On: Agar.io for Continual Reinforcement Learning](https://mlsanity.com/paper/2505.18347/) (2025)
- [A Continual Offline Reinforcement Learning Benchmark for Navigation Tasks](https://mlsanity.com/paper/2506.02883/) (2025)
- [Latent Properties of Lifelong Learning Systems](https://mlsanity.com/paper/2207.14378/) (2022)
- [Continual Reinforcement Learning with TELLA](https://mlsanity.com/paper/2208.04287/) (2022)
- [Probing for Consciousness in Machines](https://mlsanity.com/paper/2411.16262/) (2024)
- [SpecRLBench: A Benchmark for Generalization in Specification-Guided Reinforcement Learning](https://mlsanity.com/paper/2604.24729/) (2026)
- [In-Context Reinforcement Learning under Non-Stationarity: A Survey](https://mlsanity.com/paper/2607.11906/) (2026)

## Abstract (arXiv)

Progress in continual reinforcement learning has been limited due to several barriers to entry: missing code, high compute requirements, and a lack of suitable benchmarks. In this work, we present CORA, a platform for Continual Reinforcement Learning Agents that provides benchmarks, baselines, and metrics in a single code package. The benchmarks we provide are designed to evaluate different aspects of the continual RL challenge, such as catastrophic forgetting, plasticity, ability to generalize, and sample-efficient learning. Three of the benchmarks utilize video game environments (Atari, Procgen, NetHack). The fourth benchmark, CHORES, consists of four different task sequences in a visually realistic home simulator, drawn from a diverse set of task and scene parameters. To compare continual RL methods on these benchmarks, we prepare three metrics in CORA: Continual Evaluation, Isolated Forgetting, and Zero-Shot Forward Transfer. Finally, CORA includes a set of performant, open-source baselines of existing algorithms for researchers to use and expand on. We release CORA and hope that the continual RL community can benefit from our contributions, to accelerate the development of new continual RL algorithms.