mlsanityData to 14 Oct 2026

Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation

arXiv 2112.05124 ↗First posted 9 Dec 2021cs.RO cs.AI cs.CVcited by 209

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. We propose to achieve this objective by searching (via optimization) for the pose whose descriptor matches that observed in the demonstration. NDFs are conveniently trained in a self-supervised fashion via a 3D auto-encoding task that does not rely on expert-labeled keypoints. Further, NDFs are SE(3)-equivariant, guaranteeing performance that generalizes across all possible 3D object translations and rotations. We demonstrate learning of manipulation tasks from few (5-10) demonstrations both in simulation and on a real robot. Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors. Project website: https://yilundu.github.io/ndf/.

Key facts. Cited by 209 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

36 of 50 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  6. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  7. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  8. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  9. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  10. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  11. 2020Convolutional Occupancy Networkscited by 761
  12. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
24 more
  1. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  2. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 642
  3. 2020Learning Continuous Image Representation with Local Implicit Image Functioncited by 552
  4. 2021Neural Fields in Visual Computing and Beyondcited by 540
  5. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  6. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 382
  7. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  8. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  9. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 305
  10. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  11. 2020MetaSDF: Meta-learning Signed Distance Functionscited by 214
  12. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  13. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
  14. 2020Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondencecited by 119
  15. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  16. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 107
  17. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 68
  18. 2020Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Datacited by 51
  19. 2021kPAM 2.0: Feedback Control for Category-Level Robotic Manipulationcited by 38
  20. 2019kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completioncited by 27
  21. 2021Correspondence-Free Point Cloud Registration with SO(3)-Equivariant Implicit Shape Representationscited by 24
  22. 2020Self-supervised Learning for Precise Pick-and-place without Object Modelcited by 21
  23. 2021Deep Medial Fieldscited by 13
  24. 2017Pick and Place Without Geometric Object Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 33; 2023: 61; 2024: 51; 2025: 37; 2026: 27.

209 papers.

Sort:

By area:

  1. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  3. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  4. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  5. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  6. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  7. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  8. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 127
  9. 2022Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Imagescited by 121
  10. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  11. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  12. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
38 more
  1. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 81
  2. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  3. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 75
  4. 2024Equivariant Diffusion Policycited by 66
  5. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  6. 2022Zero-Shot Category-Level Object Pose Estimationcited by 57
  7. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesiscited by 57
  8. 2022Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learningcited by 56
  9. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 52
  10. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  11. 2023Geometric Algebra Transformercited by 48
  12. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
  13. 2023A new perspective on building efficient and expressive 3D equivariant graph neural networkscited by 47
  14. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 44
  15. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  16. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  17. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  18. 2022Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detectioncited by 35
  19. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  20. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  21. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  22. 2022Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrationscited by 34
  23. 2022The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetrycited by 33
  24. 2024Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3Dcited by 33
  25. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 32
  26. 2023SHACIRA: Scalable HAsh-grid Compression for Implicit Neural Representationscited by 31
  27. 2024HardNet: Hard-Constrained Neural Networks with Universal Approximation Guaranteescited by 31
  28. 2023HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulationcited by 29
  29. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  30. 2023Leveraging SE(3) Equivariance for Learning 3D Geometric Shape Assemblycited by 28
  31. 2022K-VIL: Keypoints-based Visual Imitation Learningcited by 27
  32. 2022USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulationcited by 27
  33. 2024ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matchingcited by 26
  34. 2024ET-SEED: Efficient Trajectory-Level SE(3) Equivariant Diffusion Policycited by 26
  35. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  36. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  37. 2023VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representationscited by 23
  38. 2023ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptorscited by 23

50 most cited of 209 All 209 (JSON)