mlsanityData to 7 Oct 2026

SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks

arXiv 1606.02378 ↗First posted 8 Jun 2016cs.LG cs.AI cs.CVcited by 48

Not on the map. Why?

Abstract (arXiv)

We introduce SE3-Nets, which are deep neural networks designed to model and learn rigid body motion from raw point cloud data. Based only on sequences of depth images along with action vectors and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE3 transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks. Additional experiments with a depth camera observing a Baxter robot pushing objects on a table show that SE3-Nets also work well on real data.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 13; 2023: 7; 2024: 3; 2025: 5; 2026: 3.

48 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  3. 2021Neural Articulated Radiance Fieldcited by 196
  4. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  5. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  6. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  7. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  8. 2021Weakly Supervised Learning of Rigid 3D Scene Flowcited by 60
  9. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  10. 2021Tangent Space Backpropagation for 3D Transformation Groupscited by 40
  11. 2023Geometric Clifford Algebra Networkscited by 37
  12. 2022BiFuse++: Self-supervised and Efficient Bi-projection Fusion for 360 Depth Estimationcited by 35
36 more
  1. 2021Learning Models as Functionals of Signed-Distance Fields for Manipulation Planningcited by 29
  2. 2022ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulationcited by 24
  3. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  4. 2022Unsupervised Image Representation Learning with Deep Latent Particlescited by 19
  5. 2023Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformercited by 18
  6. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  7. 2021Pushing it out of the Way: Interactive Visual Navigationcited by 14
  8. 2022ChiTransformer:Towards Reliable Stereo from Cuescited by 14
  9. 2021Self-Supervised Monocular Scene Decomposition and Depth Estimationcited by 8
  10. 2022Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approachcited by 8
  11. 2022A Review on Visual-SLAM: Advancements from Geometric Modelling to Learning-based Semantic Scene Understandingcited by 8
  12. 2021Building Affordance Relations for Robotic Agents - A Reviewcited by 7
  13. 2023Learning to Predict 3D Rotational Dynamics from Images of a Rigid Body with Unknown Mass Distributioncited by 7
  14. 2023Scalable and Efficient Continual Learning from Demonstration via a Hypernetwork-generated Stable Dynamics Modelcited by 6
  15. 2021Object and Relation Centric Representations for Push Effect Predictioncited by 5
  16. 2021Object-centric Video Prediction without Annotationcited by 5
  17. 2022Pretraining on Interactions for Learning Grounded Affordance Representationscited by 5
  18. 2022Learning Interpretable Dynamics from Images of a Freely Rotating 3D Rigid Bodycited by 4
  19. 2022Long-horizon video prediction using a dynamic latent hierarchycited by 4
  20. 2021Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddingscited by 3
  21. 2022How Far Can I Go ? : A Self-Supervised Approach for Deterministic Video Depth Forecastingcited by 3
  22. 2023Consistent Multimodal Generation via A Unified GAN Frameworkcited by 3
  23. 2025Forecasting Continuous Non-Conservative Dynamical Systems in SO(3)cited by 3
  24. 2021The Devils in the Point Clouds: Studying the Robustness of Point Cloud Convolutionscited by 2
  25. 2021GEM: Group Enhanced Model for Learning Dynamical Control Systemscited by 2
  26. 2022Collective Intelligence for 2D Push Manipulations with Mobile Robotscited by 2
  27. 2025Continuous-Time SO(3) Forecasting with Savitzky--Golay Neural Controlled Differential Equationscited by 2
  28. 2022Recognising Affordances in Predicted Futures to Plan with Consideration of Non-canonical Affordance Effectscited by 1
  29. 2022T3VIP: Transformation-based 3D Video Predictioncited by 1
  30. 2023Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Modelscited by 1
  31. 2026RigidFormer: Learning Rigid Dynamics using Transformerscited by 1
  32. 2021Towards Exploiting Geometry and Time for Fast Off-Distribution Adaptation in Multi-Task Robot Learningcited by 0
  33. 2024Deep Extrinsic Manifold Representation for Vision Taskscited by 0
  34. 2025Rigid Body Adversarial Attackscited by 0
  35. 2026Modeling What Changes: Sparse, Residual World Models for Object-Centric Manipulationcited by 0
  36. 2026World Motion Models: Flexible Sequence Modeling of SE(3) Trajectoriescited by 0