mlsanityData to 7 Oct 2026

Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement

arXiv 2307.04751 ↗First posted 10 Jul 2023cs.RO cs.CV cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

We propose a system for rearranging objects in a scene to achieve a desired object-scene placing relationship, such as a book inserted in an open slot of a bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of both scenes and objects, and is trained from demonstrations to operate directly on 3D point clouds. Our system overcomes challenges associated with the existence of many geometrically-similar rearrangement solutions for a given scene. By leveraging an iterative pose de-noising training procedure, we can fit multi-modal demonstration data and produce multi-modal outputs while remaining precise and accurate. We also show the advantages of conditioning on relevant local geometric features while ignoring irrelevant global structure that harms both generalization and precision. We demonstrate our approach on three distinct rearrangement tasks that require handling multi-modality and generalization over object shape and pose in both simulation and the real world. Project website, code, and videos: https://anthonysimeonov.github.io/rpdiff-multi-modal/

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 75 of its 94 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

75 of 94 references.

References · 75

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  11. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  12. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
38 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2020Convolutional Occupancy Networkscited by 760
  3. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  5. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  6. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  7. 2021Neural Fields in Visual Computing and Beyondcited by 539
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  9. 2021Implicit Behavioral Cloningcited by 442
  10. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  11. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  12. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 304
  13. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  14. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  15. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  16. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 275
  17. 2021How to Train Your Energy-Based Modelscited by 271
  18. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  19. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  20. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  21. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  22. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  23. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 198
  24. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 193
  25. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restorationcited by 164
  26. 2021A System for General In-Hand Object Re-Orientationcited by 155
  27. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  28. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  29. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 125
  30. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  31. 2015Human Pose Estimation with Iterative Error Feedbackcited by 121
  32. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  33. 2018A micro Lie theory for state estimation in roboticscited by 117
  34. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  35. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  36. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  37. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 75
  38. 2021Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifoldcited by 65

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 16; 2025: 8.

29 papers.

Sort:

By area:

  1. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  2. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  3. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  4. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  5. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  6. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 51
  7. 2024DexDiffuser: Generating Dexterous Grasps with Diffusion Modelscited by 46
  8. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  9. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  10. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  11. 2024Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policiescited by 18
  12. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 17
17 more
  1. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  2. 20243D-MVP: 3D Multiview Pretraining for Robotic Manipulationcited by 10
  3. 2025AnyPlace: Learning Generalized Object Placement for Robot Manipulationcited by 10
  4. 20254D Visual Pre-training for Robot Learningcited by 8
  5. 2023LVDiffusor: Distilling Functional Rearrangement Priors from Large Models into Diffusorcited by 7
  6. 2024MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policiescited by 6
  7. 2023Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Modelscited by 5
  8. 20236-DoF Stability Field via Diffusion Modelscited by 2
  9. 2024Non-rigid Relative Placement through 3D Dense Diffusioncited by 2
  10. 2024Precise Pick-and-Place using Score-Based Diffusion Networkscited by 1
  11. 2024Implicit Contact Diffuser: Sequential Contact Reasoning with Latent Point Cloud Diffusioncited by 1
  12. 2024Particle-based 6D Object Pose Estimation from Point Clouds using Diffusion Modelscited by 1
  13. 2025Simultaneous Pick and Place Detection by Combining SE(3) Diffusion Models with Differential Kinematicscited by 1
  14. 2025GOPLA: Generalizable Object Placement Learning via Synthetic Augmentation of Human Arrangementcited by 1
  15. 2024DAP: Diffusion-based Affordance Prediction for Multi-modality Storagecited by 0
  16. 2025Generating Stable Placements via Physics-guided Diffusion Modelscited by 0
  17. 2025Placeit! A Framework for Learning Robot Object Placement Skillscited by 0