mlsanityData to 7 Oct 2026

Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds

arXiv 2310.13255 ↗First posted 20 Oct 2023cs.CVcited by 45

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Recent studies have presented compelling evidence that large language models (LLMs) can equip embodied agents with the self-driven capability to interact with the world, which marks an initial step toward versatile robotics. However, these efforts tend to overlook the visual richness of open worlds, rendering the entire interactive process akin to "a blindfolded text-based game." Consequently, LLM-based agents frequently encounter challenges in intuitively comprehending their surroundings and producing responses that are easy to understand. In this paper, we propose Steve-Eye, an end-to-end trained large multimodal model designed to address this limitation. Steve-Eye integrates the LLM with a visual encoder which enables it to process visual-text inputs and generate multimodal feedback. In addition, we use a semi-automatic strategy to collect an extensive dataset comprising 850K open-world instruction pairs, empowering our model to encompass three essential functions for an agent: multimodal perception, foundational knowledge base, and skill prediction and planning. Lastly, we develop three open-world evaluation benchmarks, then carry out extensive experiments from a wide range of perspectives to validate our model's capability to strategically act and plan. Codes and datasets will be released.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 41 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

35 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
22 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  12. 2021Extract Free Dense Labels from CLIPcited by 580
  13. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  16. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  17. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  18. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  19. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  20. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  21. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  22. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 23; 2026: 8.

45 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  3. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  4. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  5. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  6. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  7. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  8. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  9. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  10. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  11. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  12. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
33 more
  1. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  2. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  3. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  4. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
  5. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  6. 2025EduPlanner: LLM-Based Multi-Agent Systems for Customized and Intelligent Instructional Designcited by 18
  7. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  8. 2026From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systemscited by 17
  9. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  10. 2024UniCode: Learning a Unified Codebook for Multimodal Large Language Modelscited by 14
  11. 2026EvoRoute: Experience-Driven Self-Routing LLM Agent Systemscited by 13
  12. 2024Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?cited by 11
  13. 2024VideoGameBunny: Towards vision assistants for video gamescited by 11
  14. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 9
  15. 2026AdaWorldPolicy: World-Model-Driven Diffusion Policy with Online Adaptive Learning for Robotic Manipulationcited by 9
  16. 2025AnimeGamer: Infinite Anime Life Simulation with Next Game State Predictioncited by 7
  17. 2025Parallelized Planning-Acting for Efficient LLM-based Multi-Agent Systems in Minecraftcited by 5
  18. 2025Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learningcited by 5
  19. 2025DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Expertscited by 5
  20. 2025See, Think, Act: Online Shopper Behavior Simulation with VLM Agentscited by 5
  21. 2024QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worldscited by 4
  22. 2025GenSwarm: Scalable Multi-Robot Code-Policy Generation and Deployment via Language Modelscited by 4
  23. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  24. 2026Experience Transfer for Multimodal LLM Agents in Minecraft Gamecited by 4
  25. 2025RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policycited by 3
  26. 2025OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesiscited by 3
  27. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  28. 2025Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilitiescited by 1
  29. 2025SEAL: Self-Evolving Agentic Learning for Conversational Question Answering over Knowledge Graphscited by 1
  30. 2026Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-usecited by 1
  31. 2026Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agentscited by 1
  32. 2026EgoActor: Grounding Task Planning into Spatial-aware Egocentric Actions for Humanoid Robots via Visual-Language Modelscited by 0
  33. 2026Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perceptioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023See and Think: Embodied Agent in Virtual Environment
  2. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
  3. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
  4. 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
  5. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
  6. 2026VirtualEnv: A Platform for Embodied AI Research
  7. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
  8. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agents