# Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning

arXiv: https://arxiv.org/abs/2206.08321 · HTML: https://mlsanity.com/paper/2206.08321/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 85 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [Unsupervised Learning of Object Keypoints for Perception and Control](https://mlsanity.com/paper/1906.11883/) (2019)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [SO(2)-Equivariant Reinforcement Learning](https://mlsanity.com/paper/2203.04439/) (2022)
- [Independent SE(3)-Equivariant Models for End-to-End Rigid Protein Docking](https://mlsanity.com/paper/2111.07786/) (2021)
- [Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks](https://mlsanity.com/paper/2012.03385/) (2020)
- [Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge](https://mlsanity.com/paper/1609.09475/) (2016)
- [RGCNN: Regularized Graph CNN for Point Cloud Segmentation](https://mlsanity.com/paper/1806.02952/) (2018)
- [Cooperative Training of Descriptor and Generator Networks](https://mlsanity.com/paper/1609.09408/) (2016)
- [Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification](https://mlsanity.com/paper/2004.01301/) (2020)
- [Learning Descriptor Networks for 3D Shape Synthesis and Analysis](https://mlsanity.com/paper/1804.00586/) (2018)
- [Equivariant Transporter Network](https://mlsanity.com/paper/2202.09400/) (2022)
- [Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation](https://mlsanity.com/paper/2111.00190/) (2021)
- [A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model](https://mlsanity.com/paper/2205.06924/) (2022)

## Built on by (most cited first, 30 of 56)

- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning](https://mlsanity.com/paper/2407.01479/) (2024)
- [Equivariant Diffusion Policy](https://mlsanity.com/paper/2407.01812/) (2024)
- [Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation](https://mlsanity.com/paper/2309.02685/) (2023)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [Diffusion Models for Robotic Manipulation: A Survey](https://mlsanity.com/paper/2504.08438/) (2025)
- [EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulation](https://mlsanity.com/paper/2310.16050/) (2023)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D](https://mlsanity.com/paper/2401.12046/) (2024)
- [HardNet: Hard-Constrained Neural Networks with Universal Approximation Guarantees](https://mlsanity.com/paper/2410.10807/) (2024)
- [Leveraging SE(3) Equivariance for Learning 3D Geometric Shape Assembly](https://mlsanity.com/paper/2309.06810/) (2023)
- [ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matching](https://mlsanity.com/paper/2409.04576/) (2024)
- [SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation](https://mlsanity.com/paper/2310.16838/) (2023)
- [Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation](https://mlsanity.com/paper/2302.03573/) (2023)
- [A General Theory of Correct, Incorrect, and Extrinsic Equivariance](https://mlsanity.com/paper/2303.04745/) (2023)
- [Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance](https://mlsanity.com/paper/2305.16314/) (2023)
- [EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervision](https://mlsanity.com/paper/2303.15440/) (2023)
- [Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies](https://mlsanity.com/paper/2406.11740/) (2024)
- [One-shot Imitation Learning via Interaction Warping](https://mlsanity.com/paper/2306.12392/) (2023)
- [A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI](https://mlsanity.com/paper/2505.01458/) (2025)
- [Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation](https://mlsanity.com/paper/2406.10615/) (2024)
- [RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation](https://mlsanity.com/paper/2403.19460/) (2024)
- [AnyPlace: Learning Generalized Object Placement for Robot Manipulation](https://mlsanity.com/paper/2502.04531/) (2025)
- [NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping](https://mlsanity.com/paper/2303.07308/) (2023)
- [SE(3)-Equivariant Robot Learning and Control: A Tutorial Survey](https://mlsanity.com/paper/2503.09829/) (2025)
- [Denoising Heat-inspired Diffusion with Insulators for Collision Free Motion Planning](https://mlsanity.com/paper/2310.12609/) (2023)
- [EqvAfford: SE(3) Equivariance for Point-Level Affordance Learning](https://mlsanity.com/paper/2408.01953/) (2024)
- [MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policies](https://mlsanity.com/paper/2409.15517/) (2024)
- [SE(3) Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation](https://mlsanity.com/paper/2411.07326/) (2024)
- [Constraint-Preserving Data Generation for Visuomotor Policy Learning](https://mlsanity.com/paper/2508.03944/) (2025)

## Abstract (arXiv)

End-to-end learning for visual robotic manipulation is known to suffer from sample inefficiency, requiring large numbers of demonstrations. The spatial roto-translation equivariance, or the SE(3)-equivariance can be exploited to improve the sample efficiency for learning robotic manipulation. In this paper, we present SE(3)-equivariant models for visual robotic manipulation from point clouds that can be trained fully end-to-end. By utilizing the representation theory of the Lie group, we construct novel SE(3)-equivariant energy-based models that allow highly sample efficient end-to-end learning. We show that our models can learn from scratch without prior knowledge and yet are highly sample efficient (5~10 demonstrations are enough). Furthermore, we show that our models can generalize to tasks with (i) previously unseen target object poses, (ii) previously unseen target object instances of the category, and (iii) previously unseen visual distractors. We experiment with 6-DoF robotic manipulation tasks to validate our models' sample efficiency and generalizability. Codes are available at: https://github.com/tomato1mule/edf