mlsanityData to 7 Oct 2026

MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control

arXiv 2403.12037 ↗First posted 18 Mar 2024cs.CVcited by 52

On the map: World models → World models for language agents

Abstract (arXiv)

It is a long-lasting goal to design a generalist-embodied agent that can follow diverse instructions in human-like ways. However, existing approaches often fail to steadily follow instructions due to difficulties in understanding abstract and sequential natural language instructions. To this end, we introduce MineDreamer, an open-ended embodied agent built upon the challenging Minecraft simulator with an innovative paradigm that enhances instruction-following ability in low-level control signal generation. Specifically, MineDreamer is developed on top of recent advances in Multimodal Large Language Models (MLLMs) and diffusion models, and we employ a Chain-of-Imagination (CoI) mechanism to envision the step-by-step process of executing instructions and translating imaginations into more precise visual prompts tailored to the current state; subsequently, the agent generates keyboard-and-mouse actions to efficiently achieve these imaginations, steadily following the instructions at each step. Extensive experiments demonstrate that MineDreamer follows single and multi-step instructions steadily, significantly outperforming the best generalist agent baseline and nearly doubling its performance. Moreover, qualitative analysis of the agent's imaginative ability reveals its generalization and comprehension of the open world.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 69 of its 80 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

69 of 80 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun
  2. 2020Language Models are Few-Shot Learners28 May

Other references · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2016Layer Normalizationcited by 5,579
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  13. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  14. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  15. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  16. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  17. 2022Visual Prompt Tuningcited by 1,674
  18. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  19. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  21. 2023Mastering Diverse Domains through World Modelscited by 1,193
  22. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  23. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  24. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  25. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  26. 2022A Generalist Agentcited by 802
  27. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  28. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  29. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  30. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  31. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  32. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  33. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  34. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  35. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  36. 2023Generating Images with Multimodal Language Modelscited by 259
  37. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  38. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 26; 2026: 10.

52 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  3. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Roboticscited by 125
  4. 2025Training Agents Inside of Scalable World Modelscited by 122
  5. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  6. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  8. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  9. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  10. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  11. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
38 more
  1. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  2. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  3. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  4. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 25
  5. 2025A Survey of Interactive Generative Videocited by 23
  6. 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 21
  7. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
  8. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  9. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  10. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  11. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 17
  12. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  13. 2025Solving New Tasks by Adapting Internet Video Knowledgecited by 13
  14. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  15. 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editingcited by 12
  16. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  17. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
  18. 2025PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AIcited by 10
  19. 2024AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generationcited by 9
  20. 2024Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Modelscited by 9
  21. 2024MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memorycited by 8
  22. 2025CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusioncited by 8
  23. 2025Open-World Skill Discovery from Unsegmented Demonstrationscited by 7
  24. 2026SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Roboticscited by 6
  25. 2024Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Modelscited by 5
  26. 2025Parallelized Planning-Acting for Efficient LLM-based Multi-Agent Systems in Minecraftcited by 5
  27. 2025MathSticks: A Benchmark for Visual Symbolic Compositional Reasoning with Matchstick Puzzlescited by 4
  28. 2025RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policycited by 3
  29. 2026Requesting Expert Reasoning: Augmenting LLM Agents with Learned Collaborative Interventioncited by 3
  30. 2026A Definition and Roadmap for World Modelscited by 3
  31. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  32. 2026Self-evolving Embodied AIcited by 2
  33. 2026AI for Games in the Foundation Model Eracited by 2
  34. 2024SC-Phi2: A Fine-tuned Small Language Model for StarCraft II Macromanagement Taskscited by 1
  35. 2025AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing?cited by 1
  36. 2024STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraftcited by 0
  37. 2025BAR: A Backward Reasoning based Agent for Complex Minecraft Taskscited by 0
  38. 2026WISE: A Long-Horizon Agent in Minecraft with Why-Which Reasoningcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Creative Agents: Empowering Agents with Imagination for Creative Tasks
  2. 2026CEDAR: Automata as Verifiable Interfaces for Language-Guided Embodied Action
  3. 2025BeliefNest: A Joint Action Simulator for Embodied Agents with Theory of Mind
  4. 2025CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter
  5. 2024ADAM: An Embodied Causal Agent in Open-World Environments
  6. 2023See and Think: Embodied Agent in Virtual Environment
  7. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
  8. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning