mlsanityData to 7 Oct 2026

DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics

arXiv 2210.02438 ↗First posted 5 Oct 2022cs.RO cs.CV cs.LGcited by 105

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We introduce the first work to explore web-scale diffusion models for robotics. DALL-E-Bot enables a robot to rearrange objects in a scene, by first inferring a text description of those objects, then generating an image representing a natural, human-like arrangement of those objects, and finally physically arranging the objects according to that goal image. We show that this is possible zero-shot using DALL-E, without needing any further example arrangements, data collection, or training. DALL-E-Bot is fully autonomous and is not restricted to a pre-defined set of objects or scenes, thanks to DALL-E's web-scale pre-training. Encouraging real-world results, with both human studies and objective metrics, show that integrating web-scale diffusion models into robotics pipelines is a promising direction for scalable, unsupervised robot learning.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 47 of its 71 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

47 of 71 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2017Mask R-CNNcited by 7,259
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
35 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  8. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  13. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  14. 2021ImageNet-21K Pretraining for the Massescited by 607
  15. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  16. 2019Learning Latent Plans from Playcited by 310
  17. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  18. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  19. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  20. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  21. 2020TResNet: High Performance GPU-Dedicated Architecturecited by 119
  22. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  23. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  24. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  25. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  26. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  27. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  28. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  29. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  30. 2021My House, My Rules: Learning Tidying Preferences with Graph Neural Networkscited by 27
  31. 2021Efficient and Interpretable Robot Manipulation with Graph Neural Networkscited by 21
  32. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangementcited by 21
  33. 2022Transformers are Adaptable Task Plannerscited by 15
  34. 2016Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertaintycited by 14
  35. 2020Autonomous Planning Based on Spatial Concepts to Tidy Up Home Environments with Service Robotscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 30; 2024: 42; 2025: 26; 2026: 5.

105 papers.

Sort:

By area:

  1. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  3. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  4. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  5. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  6. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  7. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  8. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Modelscited by 155
  9. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  10. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  11. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  12. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
38 more
  1. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Modelscited by 87
  2. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  3. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  4. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  5. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  6. 2023Language Models as Zero-Shot Trajectory Generatorscited by 59
  7. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  8. 2024AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policiescited by 50
  9. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
  10. 2023CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulationcited by 41
  11. 2024Mora: Enabling Generalist Video Generation via A Multi-Agent Frameworkcited by 41
  12. 2023M2T2: Multi-Task Masked Transformer for Object-centric Pick and Placecited by 40
  13. 2024Potential Based Diffusion Motion Planningcited by 40
  14. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  15. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 38
  16. 2023Boosting Continuous Control with Consistency Policycited by 36
  17. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  18. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  19. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35
  20. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  21. 2023Grasp-Anything: Large-scale Grasp Dataset from Foundation Modelscited by 33
  22. 2024Motion Planning Diffusion: Learning and Adapting Robot Motion Planning with Diffusion Modelscited by 32
  23. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  24. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 28
  25. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  26. 2023SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphscited by 24
  27. 2024TacDiffusion: Force-domain Diffusion Policy for Precise Tactile Manipulationcited by 23
  28. 2024Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Modelscited by 22
  29. 2024Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RLcited by 21
  30. 2024HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awarenesscited by 21
  31. 2024Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloningcited by 19
  32. 2025Towards Hierarchical Rectified Flowcited by 19
  33. 2025Sequential Diffusion Language Modelscited by 19
  34. 2023Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?cited by 18
  35. 2024Generative Image as Action Modelscited by 18
  36. 2023Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Modelscited by 17
  37. 2024Composable Part-Based Manipulationcited by 17
  38. 2024Semantically Controllable Augmentations for Generalizable Robot Learningcited by 17

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Pixel Motion Diffusion is What We Need for Robot Control
  2. 2025Tenma: Robust Cross-Embodiment Robot Manipulation with Diffusion Transformer
  3. 2025S2-Diffusion: Generalizing from Instance-level to Category-level Skills in Robot Manipulation
  4. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations
  5. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation
  6. 2026SlotDiT: Object-Centric Representations for Diffusion Transformers
  7. 2025Mini Diffuser: Fast Multi-task Diffusion Policy Training Using Two-level Mini-batches
  8. 2026Robots as Tokens: Unified Diffusion Transformer for Coordinated Multi-Robot Trajectory Generation