mlsanityData to 7 Oct 2026

Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models

arXiv 2310.10639 ↗First posted 16 Oct 2023cs.ROcited by 264

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

If generalist robots are to operate in truly unstructured environments, they need to be able to recognize and reason about novel objects and scenarios. Such objects and scenarios might not be present in the robot's own training data. We propose SuSIE, a method that leverages an image-editing diffusion model to act as a high-level planner by proposing intermediate subgoals that a low-level controller can accomplish. Specifically, we finetune InstructPix2Pix on video data, consisting of both human videos and robot rollouts, such that it outputs hypothetical future "subgoal" observations given the robot's current observation and a language command. We also use the robot data to train a low-level goal-conditioned policy to act as the aforementioned low-level controller. We find that the high-level subgoal predictions can utilize Internet-scale pretraining and visual understanding to guide the low-level goal-conditioned policy, achieving significantly better generalization and precision than conventional language-conditioned policies. We achieve state-of-the-art results on the CALVIN benchmark, and also demonstrate robust generalization on real-world manipulation tasks, beating strong baselines that have access to privileged information or that utilize orders of magnitude more compute and training data. The project website can be found at http://rail-berkeley.github.io/susie .

Key facts. Cited by 264 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 15 of its 63 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

15 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2022Video Diffusion Modelscited by 2,280
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2023Mastering Diverse Domains through World Modelscited by 1,193
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  12. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
1 more
  1. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 68; 2025: 110; 2026: 83.

264 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  3. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  4. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  5. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  6. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  7. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  8. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  9. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  10. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  11. 2026Causal World Modeling for Robot Controlcited by 289
  12. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
38 more
  1. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  2. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  3. 2025Motus: A Unified Latent Action World Modelcited by 231
  4. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  5. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  6. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  7. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  8. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  9. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  10. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  11. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  12. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  13. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  14. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  15. 2025Unified Vision-Language-Action Modelcited by 128
  16. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  17. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  18. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  19. 2025GR-3 Technical Reportcited by 110
  20. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  21. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  22. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
  23. 2024Video as the New Language for Real-World Decision Makingcited by 86
  24. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  25. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  26. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  27. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  28. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  29. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 80
  30. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  31. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  32. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  33. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  34. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  35. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  36. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  37. 2025Vidar: Embodied Video Diffusion Model for Generalist Manipulationcited by 58
  38. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57

50 most cited of 264 All 264 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  2. 2025Zero-Shot Visual Generalization in Robot Manipulation
  3. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies
  4. 2026Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
  5. 2025π0.5: a Vision-Language-Action Model with Open-World Generalization
  6. 2026Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
  7. 2025Image Generation as a Visual Planner for Robotic Manipulation
  8. 2026World Models for Robotic Manipulation: A Survey