mlsanityData to 7 Oct 2026

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft

arXiv 2306.00937 ↗First posted 1 Jun 2023cs.AI cs.LGcited by 95

Not on the map. Why?

Cited by 34 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces a methodology, inspired by unCLIP, for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories. Using this methodology, we create an instruction-tuned Video Pretraining (VPT) model called STEVE-1, which can follow short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, reducing the need for costly human text annotations, and all for only 60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 sets a new bar for open-ended instruction-following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

51 of 70 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2016Layer Normalizationcited by 5,579
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  12. 2023Mastering Diverse Domains through World Modelscited by 1,193
  13. 2017Hindsight Experience Replaycited by 1,096
  14. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  15. 2022A Generalist Agentcited by 802
  16. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  17. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  18. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  19. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  20. 2023MusicLM: Generating Music From Textcited by 442
  21. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  22. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  23. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  24. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  25. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  26. 2022Multi-Game Decision Transformerscited by 212
  27. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  28. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  29. 2022Language Model Cascadescited by 104
  30. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  31. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  32. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  33. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
  34. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  35. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  36. 2020Planning from Pixels using Inverse Dynamics Modelscited by 36
  37. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  38. 2022Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resetscited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 43; 2025: 24; 2026: 15.

95 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  2. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  3. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  4. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  5. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  6. 2025Training Agents Inside of Scalable World Modelscited by 122
  7. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  8. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  9. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  10. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  11. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  12. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
38 more
  1. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  2. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  3. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  4. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  5. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  6. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  7. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  8. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  9. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 33
  10. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  11. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  12. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 29
  13. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  14. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  15. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  16. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  17. 2024Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigationcited by 24
  18. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policycited by 24
  19. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23
  20. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  21. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
  22. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  23. 2025EduPlanner: LLM-Based Multi-Agent Systems for Customized and Intelligent Instructional Designcited by 18
  24. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  25. 2024Zero-shot Imitation Policy via Search in Demonstration Datasetcited by 13
  26. 2024Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Modelscited by 13
  27. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  28. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  29. 2024MineStudio: A Streamlined Package for Minecraft AI Agent Developmentcited by 12
  30. 2023AI Text-to-Behavior: A Study In Steerabilitycited by 11
  31. 2024COA-GPT: Generative Pre-trained Transformers for Accelerated Course of Action Development in Military Operationscited by 11
  32. 2024Reward Machines for Deep RL in Noisy and Uncertain Environmentscited by 11
  33. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  34. 2025PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AIcited by 10
  35. 2024Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policycited by 9
  36. 2024GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agentscited by 9
  37. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraftcited by 9
  38. 2023BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Taskscited by 8

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Behavioral Cloning via Search in Embedded Demonstration Dataset
  2. 2025BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
  3. 2024ADAM: An Embodied Causal Agent in Open-World Environments
  4. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
  5. 2026CEDAR: Automata as Verifiable Interfaces for Language-Guided Embodied Action
  6. 2026PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft
  7. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Game
  8. 2025Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review