mlsanityData to 7 Oct 2026

Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents

arXiv 2102.02926 ↗First posted 4 Feb 2021cs.LG cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

There has been rapidly growing interest in meta-learning as a method for increasing the flexibility and sample efficiency of reinforcement learning. One problem in this area of research, however, has been a scarcity of adequate benchmark tasks. In general, the structure underlying past benchmarks has either been too simple to be inherently interesting, or too ill-defined to support principled analysis. In the present work, we introduce a new benchmark for meta-RL research, emphasizing transparency and potential for in-depth analysis as well as structural richness. Alchemy is a 3D video game, implemented in Unity, which involves a latent causal structure that is resampled procedurally from episode to episode, affording structure learning, online inference, hypothesis testing and action sequencing based on abstract domain knowledge. We evaluate a pair of powerful RL agents on Alchemy and present an in-depth analysis of one of these agents. Results clearly indicate a frank and specific failure of meta-learning, providing validation for Alchemy as a challenging benchmark for meta-RL. Concurrent with this report, we are releasing Alchemy as public resource, together with a suite of analysis tools and sample agent trajectories.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 43 of its 86 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

43 of 86 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  11. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
31 more
  1. 2016Learning to reinforcement learncited by 369
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  3. 2017Population Based Training of Neural Networkscited by 351
  4. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  5. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  6. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  7. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  8. 2018Unity: A General Platform for Intelligent Agentscited by 232
  9. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  10. 2020The NetHack Learning Environmentcited by 196
  11. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  12. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  13. 2018Meta-Gradient Reinforcement Learningcited by 156
  14. 2019Behaviour Suite for Reinforcement Learningcited by 129
  15. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  16. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  17. 2019Distilling Policy Distillationcited by 81
  18. 2018Kickstarting Deep Reinforcement Learningcited by 75
  19. 2019Meta reinforcement learning as task inferencecited by 75
  20. 2019Meta-learning of Sequential Strategiescited by 68
  21. 2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
  22. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  23. 2019Increasing Generality in Machine Learning through Procedural Content Generationcited by 58
  24. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  25. 2020Grounded Language Learning Fast and Slowcited by 54
  26. 2019Causal Reasoning from Meta-reinforcement Learningcited by 53
  27. 2018ViZDoom Competitions: Playing Doom from Pixelscited by 50
  28. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 43
  29. 2020Meta-learning in natural and artificial intelligencecited by 39
  30. 2020Meta-trained agents implement Bayes-optimal agentscited by 37
  31. 2019Guided Meta-Policy Searchcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 11; 2024: 7; 2025: 7; 2026: 4.

35 papers.

Sort:

By area:

  1. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  2. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  3. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  4. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 37
  5. 2024The broader spectrum of in-context learningcited by 33
  6. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  7. 2023Causal Reinforcement Learning: A Surveycited by 22
  8. 2021Classical Planning in Deep Latent Spacecited by 21
  9. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  10. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  11. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  12. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
23 more
  1. 2024Does learning the right latent variables necessarily improve in-context learning?cited by 12
  2. 2022Interactive Visual Reasoning under Uncertaintycited by 8
  3. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 7
  4. 2025Benchmarking World-Model Learning with Environment-Level Queriescited by 7
  5. 2022Giving Feedback on Interactive Student Programs with Meta-Explorationcited by 6
  6. 2024IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learningcited by 6
  7. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learningcited by 5
  8. 2022A model-based approach to meta-Reinforcement Learning: Transformers and tree searchcited by 3
  9. 2024Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversitycited by 3
  10. 2022Actively learning to learn causal relationshipscited by 2
  11. 2023Learning How to Infer Partial MDPs for In-Context Adaptation and Explorationcited by 2
  12. 2023AdCraft: An Advanced Reinforcement Learning Benchmark Environment for Search Engine Marketing Optimizationcited by 2
  13. 2023Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rulescited by 2
  14. 2023Procedural generation of meta-reinforcement learning taskscited by 1
  15. 2025Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisionscited by 1
  16. 2025Understanding the Staged Dynamics of Transformers in Learning Latent Structurecited by 1
  17. 2025Robust Agents in Open-Ended Worldscited by 1
  18. 2023RL3: Boosting Meta Reinforcement Learning via RL inside RL2cited by 0
  19. 2023Hypothesis Network Planned Exploration for Rapid Meta-Reinforcement Learning Adaptationcited by 0
  20. 2026Factorization Regret mediates compositional generalization in latent spacecited by 0
  21. 2026Data Deletion Can Help in Adaptive RLcited by 0
  22. 2026Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learningcited by 0
  23. 2026Dynamic Context Scheduling: Learning Beyond the Static Universecited by 0