mlsanityData to 14 Oct 2026

Scaling Instructable Agents Across Many Simulated Worlds

arXiv 2404.10179 ↗First posted 13 Mar 2024cs.RO cs.AI cs.HCcited by 71

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building embodied AI systems that can follow arbitrary language instructions in any 3D environment is a key challenge for creating general AI. Accomplishing this goal requires learning to ground language in perception and embodied actions, in order to accomplish complex tasks. The Scalable, Instructable, Multiworld Agent (SIMA) project tackles this by training agents to follow free-form instructions across a diverse range of virtual 3D environments, including curated research environments as well as open-ended, commercial video games. Our goal is to develop an instructable agent that can accomplish anything a human can do in any simulated 3D environment. Our approach focuses on language-driven generality while imposing minimal assumptions. Our agents interact with environments in real-time using a generic, human-like interface: the inputs are image observations and language instructions and the outputs are keyboard-and-mouse actions. This general approach is challenging, but it allows agents to ground language across many visually complex and semantically rich environments while also allowing us to readily run agents in new environments. In this paper we describe our motivation and goal, the initial progress we have made, and promising preliminary results on several diverse research environments and a variety of commercial video games.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 74 of its 92 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

74 of 92 references.

References · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2023Mastering Diverse Domains through World Modelscited by 1,253
38 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2020Mastering Atari with Discrete World Modelscited by 1,007
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  11. 2022A Generalist Agentcited by 808
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  13. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  14. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  15. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  16. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  17. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  18. 2023Learning Interactive Real-World Simulatorscited by 458
  19. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  20. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  21. 2023Language Models can Solve Computer Taskscited by 372
  22. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  23. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  24. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  25. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  26. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  27. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  28. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  29. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  30. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  31. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  32. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  33. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  34. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  35. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  36. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  37. 2023Compositional Foundation Models for Hierarchical Planningcited by 95
  38. 2023Stay on topic with Classifier-Free Guidancecited by 91

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 25; 2026: 20.

71 papers.

Sort:

By area:

  1. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  2. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  3. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  4. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  5. 2024Building Machines that Learn and Think with Peoplecited by 73
  6. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  7. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  8. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  9. 2024Moral Alignment for LLM Agentscited by 39
  10. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  11. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 31
  12. 2026NitroGen: An Open Foundation Model for Generalist Gaming Agentscited by 27
38 more
  1. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  2. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 24
  3. 2024Scaling Laws for Pre-training Agents and World Modelscited by 23
  4. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  5. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  6. 2024Very Large-Scale Multi-Agent Simulation in AgentScopecited by 18
  7. 2025General agents contain world modelscited by 17
  8. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  9. 2026Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Taskscited by 15
  10. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  11. 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 13
  12. 2026Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playingcited by 11
  13. 2024DreamGarden: A Designer Assistant for Growing Games from a Single Promptcited by 10
  14. 2025BIMgent: Towards Autonomous Building Modeling via Computer-use Agentscited by 10
  15. 2024Words in Motion: Extracting Interpretable Control Vectors for Motion Transformerscited by 9
  16. 2024GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agentscited by 9
  17. 2024MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilotcited by 8
  18. 2024Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Gamescited by 8
  19. 2024LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Controlcited by 6
  20. 2025A Comprehensive Review of AI Agents: Transforming Possibilities in Technology and Beyondcited by 6
  21. 2024Frontier AI Ethics: Anticipating and Evaluating the Societal Impacts of Language Model Agentscited by 5
  22. 2025Leveraging LLM Agents for Automated Video Game Testingcited by 5
  23. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  24. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4
  25. 2025Simulation Agent: A Framework for Integrating Simulation and Large Language Models for Enhanced Decision-Makingcited by 4
  26. 2025Thousand-Brains Systems: Sensorimotor Intelligence for Rapid, Robust Learning and Inferencecited by 4
  27. 2025D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AIcited by 4
  28. 2024Instruction-Driven Game Engines on Large Language Modelscited by 3
  29. 2024Instruction-Driven Game Engine: A Poker Case Studycited by 3
  30. 2025Pixels to Play: A Foundation Model for 3D Gameplaycited by 3
  31. 2025FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Gamescited by 3
  32. 2025TV2TV: A Unified Framework for Interleaved Language and Video Generationcited by 3
  33. 2026GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agentscited by 3
  34. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 3
  35. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  36. 2026Skill Drift Is Contract Violation: Proactive Maintenance for LLM Agent Skill Librariescited by 3
  37. 2026SPIKE: An Adaptive Dual Controller Framework for Cost-Efficient Long-Horizon Game Agentscited by 3
  38. 2026AI for Games in the Foundation Model Eracited by 3

50 most cited of 71 All 71 (JSON)