mlsanityData to 7 Oct 2026

Reward-Free Curricula for Training Robust World Models

arXiv 2306.09205 ↗First posted 15 Jun 2023cs.LGcited by 13

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

There has been a recent surge of interest in developing generally-capable agents that can adapt to new tasks without additional training in the environment. Learning world models from reward-free exploration is a promising approach, and enables policies to be trained using imagined experience for new tasks. However, achieving a general agent requires robustness across different environments. In this work, we address the novel problem of generating curricula in the reward-free setting to train robust world models. We consider robustness in terms of minimax regret over all environment instantiations and show that the minimax regret can be connected to minimising the maximum error in the world model across environment instances. This result informs our algorithm, WAKER: Weighted Acquisition of Knowledge across Environments for Robustness. WAKER selects environments for data collection based on the estimated error of the world model for each environment. Our experiments demonstrate that WAKER outperforms several baselines, resulting in improved robustness, efficiency, and generalisation.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 52 of its 81 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

52 of 81 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Learning General World Models in a Handful of Reward-Free Deployments23 Oct
  2. 2020Mastering Atari with Discrete World Models5 Oct
  3. 2020Planning to Explore via Self-Supervised World Models12 May
  4. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  5. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  6. 2018Recurrent World Models Facilitate Policy Evolution4 Sep
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  8. 2018Learning to Play with Intrinsically-Motivated Self-Aware Agents21 Feb

Other references · 44

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2018DeepMind Control Suitecited by 935
  8. 2022A Generalist Agentcited by 802
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
32 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  2. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  3. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  4. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  5. 2017Count-Based Exploration with Neural Density Modelscited by 255
  6. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  7. 2017Automated Curriculum Learning for Neural Networkscited by 251
  8. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  9. 2017Teacher-Student Curriculum Learningcited by 197
  10. 2020Prioritized Level Replaycited by 190
  11. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  12. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  13. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  14. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  15. 2019Active Domain Randomizationcited by 161
  16. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  17. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  18. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  19. 2021Discovering and Achieving Goals via World Modelscited by 131
  20. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  21. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  22. 2021Replay-Guided Adversarial Environment Designcited by 108
  23. 2018Model-Based Active Explorationcited by 105
  24. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 69
  25. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  26. 2020Ready Policy One: World Building Through Active Learningcited by 33
  27. 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
  28. 2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
  29. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 20
  30. 2022General Intelligence Requires Rethinking Explorationcited by 20
  31. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  32. 2020Minimax Regret Optimisation for Robust Planning in Uncertain Markov Decision Processescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 3; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  2. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  3. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  4. 2023TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfercited by 18
  5. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  6. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  7. 2025PrivilegedDreamer: Explicit Imagination of Privileged Information for Rapid Adaptation of Learned Policiescited by 3
  8. 2023Learning Curricula in Open-Ended Worldscited by 2
  9. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modelingcited by 2
  10. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Designcited by 1
  11. 2026Improving Regret Approximation for Unsupervised Dynamic Environment Generationcited by 1
  12. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
1 more
  1. 2026Efficient Exploration Is Enoughcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning
  2. 2025RLVR-World: Training World Models with Reinforcement Learning
  3. 2026Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
  4. 2023Mastering Diverse Domains through World Models
  5. 2025General agents contain world models
  6. 2024Learning World Models for Unconstrained Goal Navigation
  7. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
  8. 2026Inverting the Bellman Equation: From Q-Values to World Models