mlsanityData to 7 Oct 2026

SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields

arXiv 2211.09786 ↗First posted 17 Nov 2022cs.RO cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

We present a method for performing tasks involving spatial relations between novel object instances initialized in arbitrary poses directly from point cloud observations. Our framework provides a scalable way for specifying new tasks using only 5-10 demonstrations. Object rearrangement is formalized as the question of finding actions that configure task-relevant parts of the object in a desired alignment. This formalism is implemented in three steps: assigning a consistent local coordinate frame to the task-relevant object parts, determining the location and orientation of this coordinate frame on unseen object instances, and executing an action that brings these frames into the desired alignment. We overcome the key technical challenge of determining task-relevant local coordinate frames from a few demonstrations by developing an optimization method based on Neural Descriptor Fields (NDFs) and a single annotated 3D keypoint. An energy-based learning scheme to model the joint configuration of the objects that satisfies a desired relational task further improves performance. The method is tested on three multi-object rearrangement tasks in simulation and on a real robot. Project website, videos, and code: https://anthonysimeonov.github.io/r-ndf/

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 72 of its 83 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

72 of 83 references.

References · 72

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Mask R-CNNcited by 7,259
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  6. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  7. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  8. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  9. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  10. 2020Convolutional Occupancy Networkscited by 760
  11. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  12. 2021Neural Fields in Visual Computing and Beyondcited by 539
38 more
  1. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  2. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 381
  3. 2020Local Implicit Grid Representations for 3D Scenescited by 372
  4. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  5. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  6. 2020Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstructioncited by 321
  7. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 304
  8. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  9. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  10. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  11. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  12. 2021Vision-Only Robot Navigation in a Neural Radiance Worldcited by 153
  13. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  14. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  15. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  16. 2020Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondencecited by 119
  17. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  18. 2018A micro Lie theory for state estimation in roboticscited by 117
  19. 2021LENS: Localization enhanced by NeRF synthesiscited by 116
  20. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  21. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  22. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  23. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  24. 2020Unseen Object Instance Segmentation for Robotic Environmentscited by 85
  25. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  26. 2021Learning to Compose Visual Relationscited by 72
  27. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  28. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  29. 2022Learning Neural Acoustic Fieldscited by 65
  30. 2020Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentationcited by 64
  31. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  32. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  33. 2022Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learningcited by 54
  34. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  35. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  36. 2022Equivariant Transporter Networkcited by 44
  37. 2018FilterReg: Robust and Efficient Probabilistic Point-Set Registration using Gaussian Filter and Twist Parameterizationcited by 43
  38. 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 20; 2025: 9; 2026: 4.

51 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  3. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  4. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 72
  5. 2024Equivariant Diffusion Policycited by 61
  6. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 51
  7. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 43
  8. 2023Compositional Diffusion-Based Continuous Constraint Solverscited by 39
  9. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  10. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  11. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  12. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
38 more
  1. 2023Leveraging SE(3) Equivariance for Learning 3D Geometric Shape Assemblycited by 27
  2. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  3. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  4. 2023Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulationcited by 21
  5. 2023Lifelong Robot Learning with Human Assisted Language Plannerscited by 19
  6. 2023A General Theory of Correct, Incorrect, and Extrinsic Equivariancecited by 18
  7. 2024Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policiescited by 18
  8. 2023One-shot Imitation Learning via Interaction Warpingcited by 16
  9. 2024Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Taskscited by 16
  10. 2024NeRFs in Robotics: A Surveycited by 16
  11. 2023HyP-NeRF: Learning Improved NeRF Priors using a HyperNetworkcited by 14
  12. 2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignmentcited by 14
  13. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  14. 2024RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentationcited by 12
  15. 2023NeuSE: Neural SE(3)-Equivariant Embedding for Consistent Spatial Understanding with Objectscited by 10
  16. 2025AnyPlace: Learning Generalized Object Placement for Robot Manipulationcited by 10
  17. 2024"Set It Up!": Functional Object Arrangement with Compositional Generative Modelscited by 9
  18. 2023NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptorscited by 8
  19. 2025SE(3)-Equivariant Robot Learning and Control: A Tutorial Surveycited by 8
  20. 2024EqvAfford: SE(3) Equivariance for Point-Level Affordance Learningcited by 6
  21. 2024MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policiescited by 6
  22. 2025Constraint-Preserving Data Generation for Visuomotor Policy Learningcited by 6
  23. 2024Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Priorcited by 5
  24. 2024Neural Attention Field: Emerging Point Relevance in 3D Scenes for One-Shot Dexterous Graspingcited by 5
  25. 2025A Practical Guide for Incorporating Symmetry in Diffusion Policycited by 4
  26. 2025EfficientFlow: Efficient Equivariant Flow Policy Learning for Embodied AIcited by 4
  27. 2024Visual Imitation Learning of Task-Oriented Object Grasping and Rearrangementcited by 3
  28. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  29. 2023"Tidy Up the Table": Grounding Common-sense Objective for Tabletop Object Rearrangementcited by 1
  30. 2023Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworkscited by 1
  31. 2024Geometric Algebra Meets Large Language Models: Instruction-Based Transformations of Separate Meshes in 3D, Interactive and Controllable Scenescited by 1
  32. 2024Implicit Contact Diffuser: Sequential Contact Reasoning with Latent Point Cloud Diffusioncited by 1
  33. 2024DAP: Diffusion-based Affordance Prediction for Multi-modality Storagecited by 0
  34. 2025GIFT: Geometry-Induced Functional Transfer for Category-level Object Manipulationcited by 0
  35. 2025Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulationcited by 0
  36. 2026LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulationcited by 0
  37. 2026Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Modelscited by 0
  38. 2026Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentationcited by 0

50 most cited of 51 All 51 (JSON)