mlsanityData to 7 Oct 2026

Prioritized Level Replay

arXiv 2010.03934 ↗First posted 8 Oct 2020cs.LG cs.AIcited by 190

Not on the map. Why?

Cited by 21 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Environments with procedurally generated content serve as important benchmarks for testing systematic generalization in deep reinforcement learning. In this setting, each level is an algorithmically created environment instance with a unique configuration of its factors of variation. Training on a prespecified subset of levels allows for testing generalization to unseen levels. What can be learned from a level depends on the current policy, yet prior work defaults to uniform sampling of training levels independently of the policy. We introduce Prioritized Level Replay (PLR), a general framework for selectively sampling the next training level by prioritizing those with higher estimated learning potential when revisited in the future. We show TD-errors effectively estimate a level's future learning potential and, when used to guide the sampling procedure, induce an emergent curriculum of increasingly difficult levels. By adapting the sampling of training levels, PLR significantly improves sample efficiency and generalization on Procgen Benchmark--matching the previous state-of-the-art in test return--and readily combines with other methods. Combined with the previous leading method, PLR raises the state-of-the-art to over 76% improvement in test return relative to standard RL baselines.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 29; 2023: 36; 2024: 31; 2025: 35; 2026: 46.

190 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  3. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  4. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  5. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  6. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  7. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  8. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  9. 2021Replay-Guided Adversarial Environment Designcited by 108
  10. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
  11. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  12. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
38 more
  1. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  2. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  3. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  4. 2022Exploration via Elliptical Episodic Bonusescited by 51
  5. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
  6. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  7. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  8. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  9. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  10. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
  11. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  12. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  13. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
  14. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  15. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  16. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  17. 2023Augmenting Autotelic Agents with Large Language Modelscited by 29
  18. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  19. 2025Training a Generally Curious Agentcited by 29
  20. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
  21. 2022Hyperbolic Deep Reinforcement Learningcited by 25
  22. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  23. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  24. 2022General Intelligence Requires Rethinking Explorationcited by 20
  25. 2023Explore to Generalize in Zero-Shot RLcited by 20
  26. 2025Learning to Reason at the Frontier of Learnabilitycited by 20
  27. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  28. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 19
  29. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
  30. 2021TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLcited by 18
  31. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  32. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  33. 2024DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraftcited by 17
  34. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  35. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  36. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  37. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  38. 2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15

50 most cited of 190 All 190 (JSON)