mlsanityData to 14 Oct 2026

Powderworld: A Platform for Understanding Generalization via Rich Task Distributions

arXiv 2211.13051 ↗First posted 23 Nov 2022cs.AI cs.LGcited by 6

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

One of the grand challenges of reinforcement learning is the ability to generalize to new tasks. However, general agents require a set of rich, diverse tasks to train on. Designing a `foundation environment' for such tasks is tricky -- the ideal environment would support a range of emergent phenomena, an expressive task space, and fast runtime. To take a step towards addressing this research bottleneck, this work presents Powderworld, a lightweight yet expressive simulation environment running directly on the GPU. Within Powderworld, two motivating challenges distributions are presented, one for world-modelling and one for reinforcement learning. Each contains hand-designed test tasks to examine generalization. Experiments indicate that increasing the environment's complexity improves generalization for world models and certain reinforcement learning agents, yet may inhibit learning in high-variance environments. Powderworld aims to support the study of generalization by providing a source of diverse tasks arising from the same core rules.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 17 of its 42 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

17 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  8. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
  9. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  10. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  11. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  12. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
4 more
  1. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  2. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  3. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  4. 2019Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agentscited by 44

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 1.

6 papers.

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  2. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  3. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  4. 2023Biomaker CA: a Biome Maker project using Cellular Automatacited by 9
  5. 2023Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Modelscited by 6
  6. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2021Procedural Generalization by Planning with Self-Supervised World Models
  2. 2026Generalization in Online Reinforcement Learning for Mobile Agents
  3. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agents
  4. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learning
  5. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learning
  6. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learning
  7. 2024Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations
  8. 2023Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards