mlsanityData to 7 Oct 2026

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

arXiv 2211.04604 ↗First posted 8 Nov 2022cs.RO cs.AI cs.CLcited by 49

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Robots operating in human environments must be able to rearrange objects into semantically-meaningful configurations, even if these objects are previously unseen. In this work, we focus on the problem of building physically-valid structures without step-by-step instructions. We propose StructDiffusion, which combines a diffusion model and an object-centric transformer to construct structures given partial-view point clouds and high-level language goals, such as "set the table". Our method can perform multiple challenging language-conditioned multi-step 3D planning tasks using one model. StructDiffusion even improves the success rate of assembling physically-valid structures out of unseen objects by on average 16% over an existing multi-modal transformer model trained on specific structures. We show experiments on held-out objects in both simulation and on real-world rearrangement tasks. Importantly, we show how integrating both a diffusion model and a collision-discriminator model allows for improved generalization over other methods when rearranging previously-unseen objects. For videos and additional results, see our website: https://structdiffusion.github.io/.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 52 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

40 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics5 Oct
  2. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion8 Sep

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2020PCT: Point cloud transformercited by 730
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
26 more
  1. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  2. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  4. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  5. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  6. 2019Learning Latent Plans from Playcited by 310
  7. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  8. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  9. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  10. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  11. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  12. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  13. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  14. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  15. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  16. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  17. 2021Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordancescited by 38
  18. 2021NeRP: Neural Rearrangement Planning for Unknown Objectscited by 33
  19. 2021Grounding Predicates through Actionscited by 25
  20. 2021NViSII: A Scriptable Tool for Photorealistic Image Generationcited by 24
  21. 2021Predicting Stable Configurations for Semantic Placement of Novel Objectscited by 23
  22. 2020A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objectscited by 22
  23. 2021FaSTrack: a Modular Framework for Real-Time Motion Planning and Guaranteed Safe Trackingcited by 15
  24. 2021Learning Perceptual Concepts by Bootstrapping from Human Queriescited by 12
  25. 2020Relational Learning for Skill Preconditionscited by 8
  26. 2020Towards Robotic Assembly by Predicting Robust, Precise and Task-oriented Graspscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 21; 2025: 11; 2026: 6.

49 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2024RoboDreamer: Learning Compositional World Models for Robot Imaginationcited by 187
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  4. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  5. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Playcited by 48
  6. 2024DexDiffuser: Generating Dexterous Grasps with Diffusion Modelscited by 46
  7. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  8. 2023Compositional Diffusion-Based Continuous Constraint Solverscited by 39
  9. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  10. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  11. 2023Adaptive Online Replanning with Diffusion Modelscited by 33
  12. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
37 more
  1. 2025GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Trainingcited by 32
  2. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  3. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  4. 2023DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observabilitycited by 21
  5. 2024Deep SE(3)-Equivariant Geometric Reasoning for Precise Placement Taskscited by 19
  6. 2024Generative Image as Action Modelscited by 18
  7. 2024Composable Part-Based Manipulationcited by 17
  8. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  9. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  10. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  11. 2024LLM-enhanced Scene Graph Learning for Household Rearrangementcited by 12
  12. 2024"Set It Up!": Functional Object Arrangement with Compositional Generative Modelscited by 9
  13. 2023LVDiffusor: Distilling Functional Rearrangement Priors from Large Models into Diffusorcited by 7
  14. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  15. 2025MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulationcited by 7
  16. 2024Learning Diffusion Policies from Demonstrations For Compliant Contact-rich Manipulationcited by 6
  17. 2023Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Modelscited by 5
  18. 2024Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Priorcited by 5
  19. 2024HACMan++: Spatially-Grounded Motion Primitives for Manipulationcited by 5
  20. 2023SceneScore: Learning a Cost Function for Object Arrangementcited by 4
  21. 2025Cascaded Diffusion Models for Neural Motion Planningcited by 4
  22. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  23. 2026ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulationcited by 3
  24. 20236-DoF Stability Field via Diffusion Modelscited by 2
  25. 2024GFPack++: Improving 2D Irregular Packing by Learning Gradient Field with Attentioncited by 2
  26. 2025MonoSE(3)-Diffusion: A Monocular SE(3) Diffusion Framework for Robust Camera-to-Robot Pose Estimationcited by 2
  27. 2023"Tidy Up the Table": Grounding Common-sense Objective for Tabletop Object Rearrangementcited by 1
  28. 2025Tidiness Score-Guided Monte Carlo Tree Search for Visual Tabletop Rearrangementcited by 1
  29. 2025Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Modelscited by 1
  30. 2026MUFASA: A Multi-Layer Framework for Slot Attentioncited by 1
  31. 2026Closing the Loop in Humanoid VLA: Persistent 3D Object Tokens for Verifiable Loco-Manipulationcited by 1
  32. 2024Sketch-Plan-Generalize: Learning and Planning with Neuro-Symbolic Programmatic Representations for Inductive Spatial Conceptscited by 0
  33. 2024Stimulating Imagination: Towards General-purpose "Something Something Placement"cited by 0
  34. 2024PACA: Perspective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangementcited by 0
  35. 2026A Cross-view Fusion Framework for Robust 6-DoF Grasp Pose Estimationcited by 0
  36. 2026ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangementcited by 0
  37. 2026Test-Time Spatial Reasoning for Robot Manipulation Using Generative Real-to-Simcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024StackGen: Generating Stable Structures from Silhouettes via Diffusion
  2. 2026SceneFoundry: Generating Interactive Infinite 3D Worlds
  3. 2024Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulation
  4. 2025LayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning
  5. 2024L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objects
  6. 2026How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning
  7. 20263D Scene Graph Prediction: Generating Hierarchical Models from Partially Observed Environments
  8. 2026LMBuild: Evaluating LLM Agents for Generating Buildable and Functional Structures