mlsanityData to 7 Oct 2026

Generative Image as Action Models

arXiv 2407.07875 ↗First posted 10 Jul 2024cs.RO cs.AI cs.CLcited by 18

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock image-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to 'draw joint-actions' as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on 25 RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state-of-the-art visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 97 of its 104 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

97 of 104 references.

Closest ancestors · 23

Same direction, newest first

  1. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations6 Mar
  2. 2024Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation6 Mar
  3. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training22 Feb
  4. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representations16 Feb
  5. 2024Simple Hierarchical Planning with Diffusion5 Jan
  6. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution18 Dec
  7. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play7 Dec
  8. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  9. 2023C3DM: Constrained-Context Conditional Diffusion Models for Imitation Learning2 Nov
  10. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  11. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models13 Oct
  12. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning4 Jul
11 more
  1. 2023Extracting Reward Functions from Diffusion Models1 Jun
  2. 2023Training Diffusion Models with Reinforcement Learning22 May
  3. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  4. 2023Affordance Diffusion: Synthesizing Hand-Object Interactions21 Mar
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  6. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  8. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects8 Nov
  9. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics5 Oct
  10. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion8 Sep
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  10. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  13. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  14. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  15. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  16. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  17. 2023Adversarial Diffusion Distillationcited by 741
  18. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  19. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  20. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  21. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  22. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  23. 2023Emergent Correspondence from Image Diffusioncited by 437
  24. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  25. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  26. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  27. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  28. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  29. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  30. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
  31. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  32. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  33. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  34. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  35. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  36. 2023Video Language Planningcited by 160
  37. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  38. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  2. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  3. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  4. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  5. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  6. 2025Robot Learning from a Physical World Modelcited by 18
  7. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  8. 2025Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulationcited by 16
  9. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  10. 2025GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulationcited by 10
  11. 2025Pixel Motion Diffusion is What We Need for Robot Controlcited by 7
  12. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challengescited by 3
6 more
  1. 2026EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fieldscited by 3
  2. 2026ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Modelscited by 2
  3. 2025World Knowledge from AI Image Generation for Robot Controlcited by 1
  4. 2025ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentationcited by 1
  5. 2025Kinematics-Aware Diffusion Policy with Consistent 3D Observation and Action Space for Whole-Arm Robotic Manipulationcited by 0
  6. 2026TAP-VLA: Tactile Annotation Prompting for Vision Language Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
  2. 2025Image Generation as a Visual Planner for Robotic Manipulation
  3. 2025FreqPolicy: Efficient Flow-based Visuomotor Policy via Frequency Consistency
  4. 2025H3DP: Triply-Hierarchical Diffusion Policy for Visuomotor Learning
  5. 20253D Flow Diffusion Policy: Visuomotor Policy Learning via Generating Flow in 3D Space
  6. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills
  7. 2026Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera
  8. 2025Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion