mlsanityData to 7 Oct 2026

ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting

arXiv 2410.17856 ↗First posted 23 Oct 2024cs.CV cs.AIcited by 20

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models (VLMs) have excelled in multimodal tasks, but adapting them to embodied decision-making in open-world environments presents challenges. One critical issue is bridging the gap between discrete entities in low-level observations and the abstract concepts required for effective planning. A common solution is building hierarchical agents, where VLMs serve as high-level reasoners that break down tasks into executable sub-tasks, typically specified using language. However, language suffers from the inability to communicate detailed spatial information. We propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from past observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, supported by real-time object tracking from SAM-2. Our method unlocks the potential of VLMs, enabling them to tackle complex tasks that demand spatial reasoning. Experiments in Minecraft show that our approach enables agents to achieve previously unattainable tasks, with a 76% absolute improvement in open-world interaction performance. Codes and demos are now available on the project page: https://craftjarvis.github.io/ROCKET-1.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 44 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

33 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  3. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar

Other references · 30

  1. 2023Segment Anythingcited by 8,158
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
18 more
  1. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  2. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  3. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  4. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  5. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  6. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  7. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  8. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  9. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  10. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  11. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  12. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  13. 2023Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worldscited by 45
  14. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  15. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  16. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  17. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  18. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 7.

20 papers.

Sort:

By area:

  1. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  2. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  3. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  4. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  5. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraftcited by 20
  6. 2024MineStudio: A Streamlined Package for Minecraft AI Agent Developmentcited by 12
  7. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraftcited by 9
  8. 2025LoopNav: Benchmarking Spatial Consistency in World Modelscited by 7
  9. 2025ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignmentcited by 6
  10. 2024Preference Goal Tuning: Post-Training as Latent Control for Frozen Policiescited by 3
  11. 2025MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agentscited by 3
  12. 2025SmartSnap: Proactive Evidence Seeking for Self-Verifying Agentscited by 3
8 more
  1. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraftcited by 3
  2. 2026MAIN-VLA: Modeling Abstraction of Intention and eNvironment for Vision-Language-Action Modelscited by 2
  3. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  4. 2026AI for Games in the Foundation Model Eracited by 2
  5. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  6. 2025Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learningcited by 0
  7. 2026World2Minecraft: Occupancy-Driven Simulated Scenes Constructioncited by 0
  8. 2026GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplaycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning
  2. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments
  3. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL
  4. 2026Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World
  5. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning
  6. 2025UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model
  7. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
  8. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents