# Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation

arXiv: https://arxiv.org/abs/2302.03573 · HTML: https://mlsanity.com/paper/2302.03573/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 23 of 23)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Vision-Only Robot Navigation in a Neural Radiance World](https://mlsanity.com/paper/2110.00168/) (2021)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [iSDF: Real-Time Neural Signed Distance Fields for Robot Perception](https://mlsanity.com/paper/2204.02296/) (2022)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields](https://mlsanity.com/paper/2203.01913/) (2022)
- [Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning](https://mlsanity.com/paper/2206.08321/) (2022)
- [Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data](https://mlsanity.com/paper/2003.01835/) (2020)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation](https://mlsanity.com/paper/2102.06279/) (2021)
- [kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion](https://mlsanity.com/paper/1909.06980/) (2019)
- [Robust Change Detection Based on Neural Descriptor Fields](https://mlsanity.com/paper/2208.01014/) (2022)

## Built on by (most cited first, 21 of 21)

- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation](https://mlsanity.com/paper/2309.02685/) (2023)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D](https://mlsanity.com/paper/2401.12046/) (2024)
- [UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation](https://mlsanity.com/paper/2506.09284/) (2025)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [Composable Part-Based Manipulation](https://mlsanity.com/paper/2405.05876/) (2024)
- [One-shot Imitation Learning via Interaction Warping](https://mlsanity.com/paper/2306.12392/) (2023)
- [Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks](https://mlsanity.com/paper/2403.03270/) (2024)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation](https://mlsanity.com/paper/2406.10615/) (2024)
- [RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation](https://mlsanity.com/paper/2403.19460/) (2024)
- [AnyPlace: Learning Generalized Object Placement for Robot Manipulation](https://mlsanity.com/paper/2502.04531/) (2025)
- [NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping](https://mlsanity.com/paper/2303.07308/) (2023)
- [NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors](https://mlsanity.com/paper/2311.01530/) (2023)
- [FuncGrasp: Learning Object-Centric Neural Grasp Functions from Single Annotated Example Object](https://mlsanity.com/paper/2402.05644/) (2024)
- [Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior](https://mlsanity.com/paper/2402.15402/) (2024)
- [Implicit Contact Diffuser: Sequential Contact Reasoning with Latent Point Cloud Diffusion](https://mlsanity.com/paper/2410.16571/) (2024)
- [Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulation](https://mlsanity.com/paper/2508.11204/) (2025)
- [LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation](https://mlsanity.com/paper/2604.08475/) (2026)

## Abstract (arXiv)

A robot operating in a household environment will see a wide range of unique and unfamiliar objects. While a system could train on many of these, it is infeasible to predict all the objects a robot will see. In this paper, we present a method to generalize object manipulation skills acquired from a limited number of demonstrations, to novel objects from unseen shape categories. Our approach, Local Neural Descriptor Fields (L-NDF), utilizes neural descriptors defined on the local geometry of the object to effectively transfer manipulation demonstrations to novel objects at test time. In doing so, we leverage the local geometry shared between objects to produce a more general manipulation framework. We illustrate the efficacy of our approach in manipulating novel objects in novel poses -- both in simulation and in the real world.