# DiffuseBot: Breeding Soft Robots With Physics-Augmented Generative Diffusion Models

arXiv: https://arxiv.org/abs/2311.17053 · HTML: https://mlsanity.com/paper/2311.17053/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 38 of its 68 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation](https://mlsanity.com/paper/2203.02923/) (2022)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC](https://mlsanity.com/paper/2302.11552/) (2023)
- [Structure-based Drug Design with Equivariant Diffusion Models](https://mlsanity.com/paper/2210.13695/) (2022)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics](https://mlsanity.com/paper/1810.01054/) (2018)
- [Scalable Differentiable Physics for Learning and Control](https://mlsanity.com/paper/2007.02168/) (2020)

## Built on by (most cited first, 30 of 31)

- [Physics-Informed Diffusion Models](https://mlsanity.com/paper/2403.14404/) (2024)
- [Constrained Synthesis with Projected Diffusion Models](https://mlsanity.com/paper/2402.03559/) (2024)
- [Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning](https://mlsanity.com/paper/2412.12953/) (2024)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [Training-Free Constrained Generation With Stable Diffusion Models](https://mlsanity.com/paper/2502.05625/) (2025)
- [Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance](https://mlsanity.com/paper/2407.13842/) (2024)
- [Neuro-Symbolic Generative Diffusion Models for Physically Grounded, Robust, and Safe Generation](https://mlsanity.com/paper/2506.01121/) (2025)
- [Dynamics-Guided Diffusion Model for Sensor-less Robot Manipulator Design](https://mlsanity.com/paper/2402.15038/) (2024)
- [DittoGym: Learning to Control Soft Shape-Shifting Robots](https://mlsanity.com/paper/2401.13231/) (2024)
- [Versatile Navigation under Partial Observability via Value-guided Diffusion Policy](https://mlsanity.com/paper/2404.02176/) (2024)
- [SOPHY: Learning to Generate Simulation-Ready Objects with Physical Materials](https://mlsanity.com/paper/2504.12684/) (2025)
- [RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills](https://mlsanity.com/paper/2506.14763/) (2025)
- [Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges](https://mlsanity.com/paper/2512.04770/) (2025)
- [Learning Low-Dimensional Strain Models of Soft Robots by Looking at the Evolution of Their Shape with Application to Model-Based Control](https://mlsanity.com/paper/2411.00138/) (2024)
- [Real-Time Verification of Embodied Reasoning for Generative Skill Acquisition](https://mlsanity.com/paper/2505.11175/) (2025)
- [Self-Guided Action Diffusion](https://mlsanity.com/paper/2508.12189/) (2025)
- [Constraint-Aware Flow Matching: Decision Aligned End-to-End Training for Constrained Sampling](https://mlsanity.com/paper/2605.12754/) (2026)
- [A Design Co-Pilot for Task-Tailored Manipulators](https://mlsanity.com/paper/2509.13077/) (2025)
- [Robust Learning of Diffusion Models with Extremely Noisy Conditions](https://mlsanity.com/paper/2510.10149/) (2025)
- [CADKnitter: Compositional CAD Generation from Text and Geometry Guidance](https://mlsanity.com/paper/2512.11199/) (2025)
- [MechStyle: Augmenting Generative AI with Mechanical Simulation to Create Stylized and Structurally Viable 3D Models](https://mlsanity.com/paper/2509.20571/) (2025)
- [BRIDGE: An Open-Source Humanoid Platform via Morphology-Control Co-Design for Physical AI](https://mlsanity.com/paper/2609.03497/) (2026)
- [Creation of Novel Soft Robot Designs using Generative AI](https://mlsanity.com/paper/2405.01824/) (2024)
- [Solving Vision Tasks with Simple Photoreceptors Instead of Cameras](https://mlsanity.com/paper/2406.11769/) (2024)
- [ProDapt: Proprioceptive Adaptation using Long-term Memory Diffusion](https://mlsanity.com/paper/2503.00193/) (2025)
- [Universal Physics Simulation: A Foundational Diffusion Approach](https://mlsanity.com/paper/2507.09733/) (2025)
- [RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Models](https://mlsanity.com/paper/2601.11801/) (2026)
- [THREAD: Trajectory Planning for Hybrid Rigid-Soft Manipulators with Environment-Aware Diffusion](https://mlsanity.com/paper/2606.21792/) (2026)
- [SoRoMoX: Fast, Differentiable, and Parallelizable Soft Robot Models](https://mlsanity.com/paper/2608.06650/) (2026)

## Abstract (arXiv)

Nature evolves creatures with a high complexity of morphological and behavioral intelligence, meanwhile computational methods lag in approaching that diversity and efficacy. Co-optimization of artificial creatures' morphology and control in silico shows promise for applications in physical soft robotics and virtual character creation; such approaches, however, require developing new learning algorithms that can reason about function atop pure structure. In this paper, we present DiffuseBot, a physics-augmented diffusion model that generates soft robot morphologies capable of excelling in a wide spectrum of tasks. DiffuseBot bridges the gap between virtually generated content and physical utility by (i) augmenting the diffusion process with a physical dynamical simulation which provides a certificate of performance, and (ii) introducing a co-design procedure that jointly optimizes physical design and control by leveraging information about physical sensitivities from differentiable simulation. We showcase a range of simulated and fabricated robots along with their capabilities. Check our website at https://diffusebot.github.io/