mlsanityData to 14 Oct 2026

See and Think: Embodied Agent in Virtual Environment

arXiv 2311.15209 ↗First posted 26 Nov 2023cs.AIcited by 54

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have achieved impressive pro-gress on several open-world tasks. Recently, using LLMs to build embodied agents has been a hotspot. This paper proposes STEVE, a comprehensive and visionary embodied agent in the Minecraft virtual environment. STEVE comprises three key components: vision perception, language instruction, and code action. Vision perception involves interpreting visual information in the environment, which is then integrated into the LLMs component with agent state and task instruction. Language instruction is responsible for iterative reasoning and decomposing complex tasks into manageable guidelines. Code action generates executable skill actions based on retrieval in skill database, enabling the agent to interact effectively within the Minecraft environment. We also collect STEVE-21K dataset, which includes 600+ vision-environment pairs, 20K knowledge question-answering pairs, and 200+ skill-code pairs. We conduct continuous block search, knowledge question and answering, and tech tree mastery to evaluate the performance. Extensive experiments show that STEVE achieves at most 1.5x faster unlocking key tech trees and 2.5x quicker in block search tasks.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 57 of its 79 references that are in the map. First posted on arXiv 26 Nov 2023.

Builds on

57 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 56

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2023Mastering Diverse Domains through World Modelscited by 1,253
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  3. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  11. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  13. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  14. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  15. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  16. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  17. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  18. 2017Embodied Question Answeringcited by 336
  19. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  20. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
  21. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  22. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  23. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  24. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  25. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  26. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 133
  27. 2019Learning Exploration Policies for Navigationcited by 116
  28. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  29. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  30. 2019Multi-Target Embodied Question Answeringcited by 74
  31. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  32. 2021VTNet: Visual Transformer Network for Object Goal Navigationcited by 51
  33. 2023Renderable Neural Radiance Map for Visual Navigationcited by 51
  34. 2022Sound Adversarial Audio-Visual Navigationcited by 48
  35. 2020See, Hear, Explore: Curiosity via Audio-Visual Associationcited by 43
  36. 2018End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learningcited by 35
  37. 2022Episodic Memory Question Answeringcited by 32
  38. 2021Towards Distraction-Robust Active Visual Trackingcited by 30

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 25; 2026: 11.

54 papers.

Sort:

By area:

  1. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  2. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  3. 2024Large Multimodal Agents: A Surveycited by 105
  4. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  5. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  6. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  7. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 59
  8. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 46
  9. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  10. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 38
  11. 2025BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attackscited by 38
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
38 more
  1. 2025JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restorationcited by 34
  2. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 33
  3. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  4. 2025Adaptive Graph Pruning for Multi-Agent Communicationcited by 29
  5. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agentscited by 28
  6. 2024CityCraft: A Real Crafter for 3D City Generationcited by 26
  7. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  8. 2024Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigationcited by 24
  9. 2024Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraintcited by 24
  10. 2024S-Agents: Self-organizing Agents in Open-ended Environmentscited by 22
  11. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  12. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  13. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 14
  14. 2024A Survey on Human-Centric LLMscited by 13
  15. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  16. 2024LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligencecited by 10
  17. 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8
  18. 2024Ego3DT: Tracking Every 3D Object in Ego-centric Videoscited by 7
  19. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraftcited by 6
  20. 2025BuilderBench: The Building Blocks of Intelligent Agentscited by 6
  21. 2025ESCA: Contextualizing Embodied Agents via Scene-Graph Generationcited by 6
  22. 2025Parallelized Planning-Acting for Efficient LLM-based Multi-Agent Systems in Minecraftcited by 5
  23. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  24. 2025AVA: Attentive VLM Agent for Mastering StarCraft IIcited by 5
  25. 2025Uncertainty in Action: Confidence Elicitation in Embodied Agentscited by 5
  26. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Gamecited by 5
  27. 2025Towards Pervasive Distributed Agentic Generative AI -- A State of The Artcited by 4
  28. 2025RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policycited by 3
  29. 2025Warehouse Spatial Question Answering with LLM Agentcited by 3
  30. 2026Gated Coordination for Efficient Multi-Agent Collaboration in Minecraft Gamecited by 3
  31. 2025EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environmentscited by 2
  32. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  33. 2026Test-Time Mixture of World Models for Embodied Agents in Dynamic Environmentscited by 2
  34. 2026Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agentscited by 2
  35. 2025Cowpox: Towards the Immunity of VLM-based Multi-Agent Systemscited by 1
  36. 2026RenderMem: Rendering as Spatial Memory Retrievalcited by 1
  37. 2026MIMIC-Py: An Extensible Tool for Personality-Driven Automated Game Testing with Large Language Modelscited by 1
  38. 2025BAR: A Backward Reasoning based Agent for Complex Minecraft Taskscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds
  2. 2025CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter
  3. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
  4. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
  5. 2024Improving Agent Interactions in Virtual Environments with Language Models
  6. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
  7. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
  8. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft