# Natural Environment Benchmarks for Reinforcement Learning

arXiv: https://arxiv.org/abs/1811.06032 · HTML: https://mlsanity.com/paper/1811.06032/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2018.

## Built on by (most cited first, 30 of 32)

- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels](https://mlsanity.com/paper/2101.02722/) (2021)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2204.13060/) (2022)
- [Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning](https://mlsanity.com/paper/2209.09203/) (2022)
- [DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck](https://mlsanity.com/paper/2102.13268/) (2021)
- [Procedural Generalization by Planning with Self-Supervised World Models](https://mlsanity.com/paper/2111.01587/) (2021)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [Learning Representations for Pixel-based Control: What Matters and Why?](https://mlsanity.com/paper/2111.07775/) (2021)
- [HarmonyDream: Task Harmonization Inside World Models](https://mlsanity.com/paper/2310.00344/) (2023)
- [Learning Latent Dynamic Robust Representations for World Models](https://mlsanity.com/paper/2405.06263/) (2024)
- [stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation](https://mlsanity.com/paper/2605.21800/) (2026)
- [Domain Adversarial Reinforcement Learning](https://mlsanity.com/paper/2102.07097/) (2021)
- [RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability](https://mlsanity.com/paper/2309.00082/) (2023)
- [IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning](https://mlsanity.com/paper/2407.05399/) (2024)
- [Invariant Causal Imitation Learning for Generalizable Policies](https://mlsanity.com/paper/2311.01489/) (2023)
- [Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter](https://mlsanity.com/paper/2208.03374/) (2022)
- [Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning](https://mlsanity.com/paper/2502.19652/) (2025)
- [Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories](https://mlsanity.com/paper/2304.13424/) (2023)
- [iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning](https://mlsanity.com/paper/2406.02696/) (2024)
- [Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hex](https://mlsanity.com/paper/2211.14673/) (2022)
- [Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments](https://mlsanity.com/paper/2106.10365/) (2021)
- [Bisimulation metric for Model Predictive Control](https://mlsanity.com/paper/2410.04553/) (2024)
- [Learning Generalizable Representations for Reinforcement Learning via Adaptive Meta-learner of Behavioral Similarities](https://mlsanity.com/paper/2212.13088/) (2022)
- [R3L: Relative Representations for Reinforcement Learning](https://mlsanity.com/paper/2404.12917/) (2024)
- [Generalization Through the Lens of Learning Dynamics](https://mlsanity.com/paper/2212.05377/) (2022)
- [Towards Control-Centric Representations in Reinforcement Learning from Images](https://mlsanity.com/paper/2310.16655/) (2023)
- [Optimal Control with Natural Images: Efficient Reinforcement Learning using Overcomplete Sparse Codes](https://mlsanity.com/paper/2412.08893/) (2024)

## Abstract (arXiv)

While current benchmark reinforcement learning (RL) tasks have been useful to drive progress in the field, they are in many ways poor substitutes for learning with real-world data. By testing increasingly complex RL algorithms on low-complexity simulation environments, we often end up with brittle RL policies that generalize poorly beyond the very specific domain. To combat this, we propose three new families of benchmark RL domains that contain some of the complexity of the natural world, while still supporting fast and extensive data acquisition. The proposed domains also permit a characterization of generalization through fair train/test separation, and easy comparison and replication of results. Through this work, we challenge the RL research community to develop more robust algorithms that meet high standards of evaluation.