mlsanityData to 7 Oct 2026

Leveraging Procedural Generation to Benchmark Reinforcement Learning

arXiv 1912.01588 ↗First posted 3 Dec 2019cs.LG stat.MLcited by 478

Not on the map. Why?

Cited by 33 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that larger models significantly improve both sample efficiency and generalization.

Key facts. Cited by 478 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 69; 2022: 85; 2023: 87; 2024: 83; 2025: 77; 2026: 76.

478 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2022A Generalist Agentcited by 802
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  5. 2021Domain Generalization: A Surveycited by 760
  6. 2024Genie: Generative Interactive Environmentscited by 706
  7. 2024Latent Action Pretraining from Videoscited by 302
  8. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  9. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  10. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  11. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  12. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
38 more
  1. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  2. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  3. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  4. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  5. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  6. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  7. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  8. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  9. 2023Learning to Act without Actionscited by 116
  10. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  11. 2021Learning to drive from a world on railscited by 114
  12. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  13. 2021Replay-Guided Adversarial Environment Designcited by 108
  14. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  15. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  16. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  17. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  18. 2022Broken Neural Scaling Lawscited by 102
  19. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  20. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  21. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  22. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  23. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  24. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  25. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  26. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  27. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  28. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 63
  29. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  30. 2022A Generalist Neural Algorithmic Learnercited by 58
  31. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  32. 2022GraphWorld: Fake Graphs Bring Real Insights for GNNscited by 56
  33. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  34. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54
  35. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  36. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 52
  37. 2022Exploration via Elliptical Episodic Bonusescited by 51
  38. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48

50 most cited of 478 All 478 (JSON)