mlsanityData to 7 Oct 2026

Benchmarking the Spectrum of Agent Capabilities

arXiv 2109.06780 ↗First posted 14 Sep 2021cs.AI cs.LGcited by 193

Not on the map. Why?

Cited by 45 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluating the general abilities of intelligent agents requires complex simulation environments. Existing benchmarks typically evaluate only one narrow task per environment, requiring researchers to perform expensive training runs on many different environments. We introduce Crafter, an open world survival game with visual inputs that evaluates a wide range of general abilities within a single environment. Agents either learn from the provided reward signal or through intrinsic objectives and are evaluated by semantically meaningful achievements that can be unlocked during each episode, such as discovering resources and crafting tools. Consistently unlocking all achievements requires strong generalization, deep exploration, and long-term reasoning. We experimentally verify that Crafter is of appropriate difficulty to drive future research and provide baselines scores of reward agents and unsupervised agents. Furthermore, we observe sophisticated behaviors emerging from maximizing the reward signal, such as building tunnel systems, bridges, houses, and plantations. We hope that Crafter will accelerate research progress by quickly evaluating a wide spectrum of abilities.

Key facts. Cited by 193 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

18 of 24 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Mastering Atari with Discrete World Modelscited by 990
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2018DeepMind Control Suitecited by 935
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  7. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  8. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  9. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  10. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  11. 2018Unity: A General Platform for Intelligent Agentscited by 232
  12. 2020The NetHack Learning Environmentcited by 196
6 more
  1. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  2. 2019Behaviour Suite for Reinforcement Learningcited by 129
  3. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  4. 2019When to use parametric models in reinforcement learning?cited by 118
  5. 2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 40
  6. 2020Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 22; 2023: 32; 2024: 37; 2025: 55; 2026: 46.

193 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  4. 2022Transformers are Sample-Efficient World Modelscited by 317
  5. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  6. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  7. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  9. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  10. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  11. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  12. 2024A Survey on Large Language Model-Based Game Agentscited by 106
38 more
  1. 2022Deep Hierarchical Planning from Pixelscited by 96
  2. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  3. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 77
  4. 2024Simplifying Deep Temporal Difference Learningcited by 71
  5. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  6. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  7. 2023Vision-Language Models as a Source of Rewardscited by 65
  8. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  9. 2022Exploration via Elliptical Episodic Bonusescited by 51
  10. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  11. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  12. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  13. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  14. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  15. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 33
  16. 2024Efficient World Models with Context-Aware Tokenizationcited by 33
  17. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  18. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  19. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  20. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  21. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  22. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  23. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  24. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  25. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 26
  26. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  27. 2022Mind the gap: Challenges of deep learning approaches to Theory of Mindcited by 21
  28. 2025Learning Transformer-based World Models with Contrastive Predictive Codingcited by 20
  29. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  30. 2024AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Makingcited by 19
  31. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agentscited by 18
  32. 2022Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worldscited by 17
  33. 2023Out of the Cage: How Stochastic Parrots Win in Cyber Security Environmentscited by 17
  34. 2023Curious Replay for Model-based Adaptationcited by 16
  35. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 16
  36. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competitioncited by 15
  37. 2025Episodic Novelty Through Temporal Distancecited by 15
  38. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14

50 most cited of 193 All 193 (JSON)