# CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning

arXiv: https://arxiv.org/abs/2110.02102 · HTML: https://mlsanity.com/paper/2110.02102/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 62 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 30 of 42)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Multi-Task Reinforcement Learning with Context-based Representations](https://mlsanity.com/paper/2102.06177/) (2021)
- [Provably efficient RL with Rich Observations via Latent State Decoding](https://mlsanity.com/paper/1901.09018/) (2019)
- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [Behaviour Suite for Reinforcement Learning](https://mlsanity.com/paper/1908.03568/) (2019)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [Hyperparameter Importance Across Datasets](https://mlsanity.com/paper/1710.04725/) (2017)
- [Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits](https://mlsanity.com/paper/2002.02518/) (2020)
- [Learning Task Informed Abstractions](https://mlsanity.com/paper/2106.15612/) (2021)
- [Markov Decision Processes with Continuous Side Information](https://mlsanity.com/paper/1711.05726/) (2017)

## Built on by (most cited first, 21 of 21)

- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Parseval Regularization for Continual Reinforcement Learning](https://mlsanity.com/paper/2412.07224/) (2024)
- [The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning](https://mlsanity.com/paper/2210.08607/) (2022)
- [Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective](https://mlsanity.com/paper/2305.18464/) (2023)
- [Reinforcement Learning in Presence of Discrete Markovian Context Evolution](https://mlsanity.com/paper/2202.06557/) (2022)
- [Generalization in Online Reinforcement Learning for Mobile Agents](https://mlsanity.com/paper/2603.07432/) (2026)
- [Out-of-Distribution Generalization with a SPARC: Racing 100 Unseen Vehicles with a Single Policy](https://mlsanity.com/paper/2511.09737/) (2025)
- [Hyperparameters in Contextual RL are Highly Situational](https://mlsanity.com/paper/2212.10876/) (2022)
- [Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking](https://mlsanity.com/paper/2510.02490/) (2025)
- [Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring](https://mlsanity.com/paper/2304.00420/) (2023)
- [Deep Reinforcement Learning with Explicit Context Representation](https://mlsanity.com/paper/2310.09924/) (2023)
- [Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies](https://mlsanity.com/paper/2512.04279/) (2025)
- [Planning for Change: Reinforcement Learning Combined with Bounded Extremum Seeking for Robotic Control under Distribution Shift](https://mlsanity.com/paper/2604.01142/) (2026)
- [PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments](https://mlsanity.com/paper/2608.14441/) (2026)
- [AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learning](https://mlsanity.com/paper/2208.02376/) (2022)
- [Dyadic Reinforcement Learning](https://mlsanity.com/paper/2308.07843/) (2023)
- [Zero-shot Imitation Learning by Latent Topology Mapping](https://mlsanity.com/paper/2605.08450/) (2026)
- [BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control](https://mlsanity.com/paper/2605.16170/) (2026)
- [Learning Top-k Subtask Planning Tree based on Discriminative Representation Pre-training for Decision Making](https://mlsanity.com/paper/2312.11027/) (2023)
- [Self Paced Gaussian Contextual Reinforcement Learning](https://mlsanity.com/paper/2603.23755/) (2026)

## Abstract (arXiv)

While Reinforcement Learning has made great strides towards solving ever more complicated tasks, many algorithms are still brittle to even slight changes in their environment. This is a limiting factor for real-world applications of RL. Although the research community continuously aims at improving both robustness and generalization of RL algorithms, unfortunately it still lacks an open-source set of well-defined benchmark problems based on a consistent theoretical framework, which allows comparing different approaches in a fair, reliable and reproducibleway. To fill this gap, we propose CARL, a collection of well-known RL environments extended to contextual RL problems to study generalization. We show the urgent need of such benchmarks by demonstrating that even simple toy environments become challenging for commonly used approaches if different contextual instances of this task have to be considered. Furthermore, CARL allows us to provide first evidence that disentangling representation learning of the states from the policy learning with the context facilitates better generalization. By providing variations of diverse benchmarks from classic control, physical simulations, games and a real-world application of RNA design, CARL will allow the community to derive many more such insights on a solid empirical foundation.