mlsanityData to 7 Oct 2026

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

arXiv 2206.08853 ↗First posted 17 Jun 2022cs.LG cs.AI cs.CLcited by 469

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite, knowledge bases, algorithm implementation, and pretrained models (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.

Key facts. Cited by 469 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 111 of its 158 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

111 of 158 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Language Models are Few-Shot Learners28 May

Other references · 110

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  13. 2020GLU Variants Improve Transformercited by 1,724
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  15. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  16. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  17. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  18. 2018Datasheets for Datasetscited by 1,371
  19. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  20. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  21. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  22. 2017Mixed Precision Trainingcited by 1,135
  23. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  24. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  25. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  26. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  27. 2018DeepMind Control Suitecited by 935
  28. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  29. 2022A Generalist Agentcited by 802
  30. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  31. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  32. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  33. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  34. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  35. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  36. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  37. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  38. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 124; 2024: 139; 2025: 113; 2026: 78.

469 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  7. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  8. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  9. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  10. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  11. 2023Language Models can Solve Computer Taskscited by 371
  12. 2023An Embodied Generalist Agent in 3D Worldcited by 355
38 more
  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  2. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  3. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  4. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  5. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  6. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  7. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  8. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  10. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  11. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  12. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  13. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  14. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  15. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  16. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  17. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  18. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  19. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  20. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  21. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  22. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  23. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  24. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  25. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  26. 2023Vision-Language Models as Success Detectorscited by 138
  27. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  28. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  29. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  30. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  31. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  32. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  33. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  34. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  35. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  36. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  37. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  38. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107

50 most cited of 469 All 469 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Training Agents Inside of Scalable World Models
  2. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
  3. 2021Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft
  4. 2026PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft
  5. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem
  6. 2022Behavioral Cloning via Search in Video PreTraining Latent Space
  7. 2025BeliefNest: A Joint Action Simulator for Embodied Agents with Theory of Mind
  8. 2025BAR: A Backward Reasoning based Agent for Complex Minecraft Tasks