# SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks

arXiv: https://arxiv.org/abs/1606.02378 · HTML: https://mlsanity.com/paper/1606.02378/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2016.

## Built on by (most cited first, 30 of 49)

- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)
- [Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction](https://mlsanity.com/paper/2103.04174/) (2021)
- [Weakly Supervised Learning of Rigid 3D Scene Flow](https://mlsanity.com/paper/2102.08945/) (2021)
- [Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery](https://mlsanity.com/paper/2105.01047/) (2021)
- [Tangent Space Backpropagation for 3D Transformation Groups](https://mlsanity.com/paper/2103.12032/) (2021)
- [Geometric Clifford Algebra Networks](https://mlsanity.com/paper/2302.06594/) (2023)
- [BiFuse++: Self-supervised and Efficient Bi-projection Fusion for 360 Depth Estimation](https://mlsanity.com/paper/2209.02952/) (2022)
- [Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning](https://mlsanity.com/paper/2110.00792/) (2021)
- [Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow](https://mlsanity.com/paper/2512.24766/) (2025)
- [ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation](https://mlsanity.com/paper/2203.06856/) (2022)
- [Unsupervised Image Representation Learning with Deep Latent Particles](https://mlsanity.com/paper/2205.15821/) (2022)
- [Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer](https://mlsanity.com/paper/2312.12467/) (2023)
- [FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation](https://mlsanity.com/paper/2505.10075/) (2025)
- [Pushing it out of the Way: Interactive Visual Navigation](https://mlsanity.com/paper/2104.14040/) (2021)
- [ChiTransformer:Towards Reliable Stereo from Cues](https://mlsanity.com/paper/2203.04554/) (2022)
- [Self-Supervised Monocular Scene Decomposition and Depth Estimation](https://mlsanity.com/paper/2110.11275/) (2021)
- [Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach](https://mlsanity.com/paper/2207.09314/) (2022)
- [A Review on Visual-SLAM: Advancements from Geometric Modelling to Learning-based Semantic Scene Understanding](https://mlsanity.com/paper/2209.05222/) (2022)
- [Building Affordance Relations for Robotic Agents - A Review](https://mlsanity.com/paper/2105.06706/) (2021)
- [Learning to Predict 3D Rotational Dynamics from Images of a Rigid Body with Unknown Mass Distribution](https://mlsanity.com/paper/2308.14666/) (2023)
- [Scalable and Efficient Continual Learning from Demonstration via a Hypernetwork-generated Stable Dynamics Model](https://mlsanity.com/paper/2311.03600/) (2023)
- [Object and Relation Centric Representations for Push Effect Prediction](https://mlsanity.com/paper/2102.02100/) (2021)
- [Object-centric Video Prediction without Annotation](https://mlsanity.com/paper/2105.02799/) (2021)
- [Pretraining on Interactions for Learning Grounded Affordance Representations](https://mlsanity.com/paper/2207.02272/) (2022)
- [Learning Interpretable Dynamics from Images of a Freely Rotating 3D Rigid Body](https://mlsanity.com/paper/2209.11355/) (2022)

## Abstract (arXiv)

We introduce SE3-Nets, which are deep neural networks designed to model and learn rigid body motion from raw point cloud data. Based only on sequences of depth images along with action vectors and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE3 transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks. Additional experiments with a depth camera observing a Baxter robot pushing objects on a table show that SE3-Nets also work well on real data.