mlsanityData to 7 Oct 2026

GROOT: Learning to Follow Instructions by Watching Gameplay Videos

arXiv 2310.08235 ↗First posted 12 Oct 2023cs.AI cs.LGcited by 42

Not on the map. Why?

Cited by 22 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space. We implement our agent GROOT in a simple yet effective encoder-decoder architecture based on causal transformers. We evaluate GROOT against open-world counterparts and human players on a proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the human-machine gap as well as exhibiting a 70% winning rate over the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis. The project page is available at https://craftjarvis-groot.github.io.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

50 of 69 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023Segment Anythingcited by 8,158
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2022A Generalist Agentcited by 802
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  12. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  13. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  14. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  15. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  16. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  17. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  18. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  19. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  20. 2019Goal-conditioned Imitation Learningcited by 159
  21. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  22. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  23. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  24. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  25. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  26. 2018Playing hard exploration games by watching YouTubecited by 98
  27. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  28. 2023MindAgent: Emergent Gaming Interactioncited by 87
  29. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  30. 2021VTNet: Visual Transformer Network for Object Goal Navigationcited by 50
  31. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreementcited by 46
  32. 2023Towards A Unified Agent with Foundation Modelscited by 46
  33. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  34. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  35. 2021JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learningcited by 28
  36. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  37. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
  38. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 19; 2025: 11; 2026: 8.

42 papers.

Sort:

By area:

  1. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  2. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  3. 2025Training Agents Inside of Scalable World Modelscited by 122
  4. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  5. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  6. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  7. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  8. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  9. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  10. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policycited by 24
  11. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  12. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
30 more
  1. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  2. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  3. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  4. 2024MineStudio: A Streamlined Package for Minecraft AI Agent Developmentcited by 12
  5. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  6. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  7. 2024Mars: Situated Inductive Reasoning in an Open-World Environmentcited by 10
  8. 2025Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agentscited by 9
  9. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraftcited by 9
  10. 2024MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memorycited by 8
  11. 2025Open-World Skill Discovery from Unsegmented Demonstrationscited by 7
  12. 2023Visual Encoders for Data-Efficient Imitation Learning in Modern Video Gamescited by 6
  13. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  14. 2025ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignmentcited by 6
  15. 2024Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verificationcited by 5
  16. 2025AVA: Attentive VLM Agent for Mastering StarCraft IIcited by 5
  17. 2024Read to Play (R2-Play): Decision Transformer with Multimodal Game Instructioncited by 4
  18. 2024Visual Grounding for Object-Level Generalization in Reinforcement Learningcited by 4
  19. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Gamecited by 4
  20. 2024Preference Goal Tuning: Post-Training as Latent Control for Frozen Policiescited by 3
  21. 2025RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policycited by 3
  22. 2025Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agentscited by 3
  23. 2026How VLAs (Really) Work In Open-World Environmentscited by 3
  24. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraftcited by 3
  25. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  26. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  27. 2024STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraftcited by 0
  28. 2026World2Minecraft: Occupancy-Driven Simulated Scenes Constructioncited by 0
  29. 2026WISE: A Long-Horizon Agent in Minecraft with Why-Which Reasoningcited by 0
  30. 2026GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplaycited by 0