mlsanityData to 14 Oct 2026

RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation

arXiv 2403.19460 ↗First posted 28 Mar 2024cs.RO cs.AIcited by 12

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We present RiEMann, an end-to-end near Real-time SE(3)-Equivariant Robot Manipulation imitation learning framework from scene point cloud input. Compared to previous methods that rely on descriptor field matching, RiEMann directly predicts the target poses of objects for manipulation without any object segmentation. RiEMann learns a manipulation task from scratch with 5 to 10 demonstrations, generalizes to unseen SE(3) transformations and instances of target objects, resists visual interference of distracting objects, and follows the near real-time pose change of the target object. The scalable action space of RiEMann facilitates the addition of custom equivariant actions such as the direction of turning the faucet, which makes articulated object manipulation possible for RiEMann. In simulation and real-world 6-DOF robot manipulation experiments, we test RiEMann on 5 categories of manipulation tasks with a total of 25 variants and show that RiEMann outperforms baselines in both task success rates and SE(3) geodesic distance errors on predicted poses (reduced by 68.6%), and achieves a 5.4 frames per second (FPS) network inference speed. Code and video results are available at https://riemann-web.github.io/.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 60 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

48 of 60 references.

References · 48

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  3. 2016Group Equivariant Convolutional Networkscited by 1,165
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  5. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 746
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  7. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 607
  8. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  11. 2020Reinforcement Learning with Augmented Datacited by 468
  12. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
36 more
  1. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 354
  2. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  3. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  4. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  5. 2016Steerable CNNscited by 305
  6. 2021Vector Neurons: A General Framework for SO(3)-Equivariant Networkscited by 305
  7. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  8. 2019Cormorant: Covariant Molecular Neural Networkscited by 263
  9. 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
  10. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 228
  11. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  12. 2021A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groupscited by 187
  13. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  14. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
  15. 2020Lorentz Group Equivariant Neural Network for Particle Physicscited by 121
  16. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  17. 2016Rotation equivariant vector field networkscited by 99
  18. 2022SO(2)-Equivariant Reinforcement Learningcited by 84
  19. 2017Polar Transformer Networkscited by 81
  20. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 78
  21. 2022Geometrically Equivariant Graph Neural Networks: A Surveycited by 56
  22. 2022Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learningcited by 56
  23. 2022Sample Efficient Grasp Learning Using Equivariant Modelscited by 53
  24. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulationcited by 52
  25. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  26. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
  27. 2022Equivariant Transporter Networkcited by 46
  28. 2022On-Robot Learning With Equivariant Modelscited by 42
  29. 2023EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulationcited by 37
  30. 2022Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detectioncited by 35
  31. 2024Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3Dcited by 33
  32. 2022USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulationcited by 27
  33. 2020Theoretical Aspects of Group Equivariant Neural Networkscited by 25
  34. 2023Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulationcited by 21
  35. 2023Leveraging Symmetries in Pick and Placecited by 19
  36. 2023EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervisioncited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 2.

12 papers.

  1. 2024ET-SEED: Efficient Trajectory-Level SE(3) Equivariant Diffusion Policycited by 26
  2. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  3. 2025Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulationcited by 13
  4. 2025AnyPlace: Learning Generalized Object Placement for Robot Manipulationcited by 11
  5. 2025GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulationcited by 11
  6. 2025SE(3)-Equivariant Robot Learning and Control: A Tutorial Surveycited by 10
  7. 2025A Practical Guide for Incorporating Symmetry in Diffusion Policycited by 5
  8. 20253D Equivariant Visuomotor Policy Learning via Spherical Projectioncited by 5
  9. 2025Task-Oriented 6-DoF Grasp Pose Detection in Clutterscited by 4
  10. 2026SAVLA: Symmetry-Aware Vision-Language-Action Models for Robotic Manipulationcited by 1
  11. 2025Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulationcited by 0
  12. 2026Screw Attention: Rigid-Body Algebra Inside a Transformercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Canonical Policy: Learning Canonical 3D Representation for SE(3)-Equivariant Policy
  2. 2025PRISM: Pointcloud Reintegrated Inference via Segmentation and Cross-attention for Manipulation
  3. 2026EquiForm: Noise-Robust SE(3)-Equivariant Policy Learning from 3D Point Clouds
  4. 2022ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds
  5. 2026Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation
  6. 2025Eq.Bot: Enhance Robotic Manipulation Learning via Group Equivariant Canonicalization
  7. 2025Improving Robotic Manipulation with Efficient Geometry-Aware Vision Encoder
  8. 2024Learning Generalizable 3D Manipulation With 10 Demonstrations