mlsanityData to 7 Oct 2026

Octopus: Embodied Vision-Language Programmer from Environmental Feedback

arXiv 2310.08588 ↗First posted 12 Oct 2023cs.CV cs.AI cs.LGcited by 73

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning. When integrated into an embodied agent, existing embodied VLM works either output detailed action sequences at the manipulation level or only provide plans at an abstract level, leaving a gap between high-level planning and real-world manipulation. To bridge this gap, we introduce Octopus, an embodied vision-language programmer that uses executable code generation as a medium to connect planning and manipulation. Octopus is designed to 1) proficiently comprehend an agent's visual and textual task objectives, 2) formulate intricate action sequences, and 3) generate executable code. To facilitate Octopus model development, we introduce OctoVerse: a suite of environments tailored for benchmarking vision-based code generators on a wide spectrum of tasks, ranging from mundane daily chores in simulators to sophisticated interactions in complex video games such as Grand Theft Auto (GTA) and Minecraft. To train Octopus, we leverage GPT-4 to control an explorative agent that generates training data, i.e., action blueprints and corresponding executable code. We also collect feedback that enables an enhanced training scheme called Reinforcement Learning with Environmental Feedback (RLEF). Through a series of experiments, we demonstrate Octopus's functionality and present compelling results, showing that the proposed RLEF refines the agent's decision-making. By open-sourcing our simulation environments, dataset, and model architecture, we aspire to ignite further innovation and foster collaborative applications within the broader embodied AI community.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 102 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

60 of 102 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  3. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2016OpenAI Gymcited by 1,684
  7. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  13. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  14. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  15. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  16. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  17. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  19. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  20. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  21. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  22. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  23. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  24. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  25. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  26. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  27. 2017Embodied Question Answeringcited by 335
  28. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  29. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  30. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  31. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  32. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  33. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  34. 2022Panoptic Scene Graph Generationcited by 114
  35. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  36. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  37. 2023Embodied Task Planning with Large Language Modelscited by 84
  38. 2020An Exploration of Embodied Visual Explorationcited by 56

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 27; 2025: 30; 2026: 10.

73 papers.

Sort:

By area:

  1. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  2. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  4. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  5. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  6. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  7. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  8. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  9. 2024Large Multimodal Agents: A Surveycited by 99
  10. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  11. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 67
  12. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
38 more
  1. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  2. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
  3. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  4. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  5. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  6. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 35
  7. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  8. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  9. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  10. 2023LARP: Language-Agent Role Play for Open-World Gamescited by 24
  11. 2024Large Language Models and Video Games: A Preliminary Scoping Reviewcited by 23
  12. 2024Efficient Inference of Vision Instruction-Following Models with Elastic Cachecited by 23
  13. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  14. 20244D Panoptic Scene Graph Generationcited by 22
  15. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  16. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  17. 2024Navigation Instruction Generation with BEV Perception and Large Language Modelscited by 20
  18. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  19. 2025Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Drivingcited by 16
  20. 2025HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronizationcited by 14
  21. 2025Hybrid Agents for Image Restorationcited by 14
  22. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  23. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 12
  24. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Modelscited by 11
  25. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 11
  26. 2025Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planningcited by 10
  27. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Plannercited by 9
  28. 2024A Robotic Skill Learning System Built Upon Diffusion Policies and Foundation Modelscited by 8
  29. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  30. 2025Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learningcited by 8
  31. 20253EED: Ground Everything Everywhere in 3Dcited by 8
  32. 2025Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Followingcited by 7
  33. 2025Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model Systemcited by 5
  34. 2025EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLMcited by 5
  35. 2025Enhancing Visual Planning with Auxiliary Tasks and Multi-token Predictioncited by 5
  36. 2025Distracted Robot: How Visual Clutter Undermine Robotic Manipulationcited by 5
  37. 2025Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspectivecited by 4
  38. 2024LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Applicationcited by 3

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
  2. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness
  3. 2023See and Think: Embodied Agent in Virtual Environment
  4. 2026CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
  5. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning
  6. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
  7. 2025CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter
  8. 2025Igniting VLMs toward the Embodied Space