# Learning with 3D rotations, a hitchhiker's guide to SO(3)

arXiv: https://arxiv.org/abs/2404.11735 · HTML: https://mlsanity.com/paper/2404.11735/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 13 of 13)

- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [An Analysis of SVD for Deep Rotation Estimation](https://mlsanity.com/paper/2006.14616/) (2020)
- [Deep Regression on Manifolds: A 3D Rotation Case Study](https://mlsanity.com/paper/2103.16317/) (2021)
- [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019)
- [A Loss Curvature Perspective on Training Instability in Deep Learning](https://mlsanity.com/paper/2110.04369/) (2021)
- [Projective Manifold Gradient Layer for Deep Rotation Regression](https://mlsanity.com/paper/2110.11657/) (2021)
- [Loss it right: Euclidean and Riemannian Metrics in Learning-based Visual Odometry](https://mlsanity.com/paper/2401.05396/) (2023)

## Built on by (most cited first, 30 of 42)

- [From Imitation to Refinement -- Residual RL for Precise Assembly](https://mlsanity.com/paper/2407.16677/) (2024)
- [Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching](https://mlsanity.com/paper/2409.07343/) (2024)
- [Towards Human-level Intelligence via Human-like Whole-Body Manipulation](https://mlsanity.com/paper/2507.17141/) (2025)
- [Flexible MOF Generation with Torsion-Aware Flow Matching](https://mlsanity.com/paper/2505.17914/) (2025)
- [Steerable Scene Generation with Post Training and Inference-Time Search](https://mlsanity.com/paper/2505.04831/) (2025)
- [Mind to Hand: Purposeful Robotic Control via Embodied Reasoning](https://mlsanity.com/paper/2512.08580/) (2025)
- [Reconstructing Animals and the Wild](https://mlsanity.com/paper/2411.18807/) (2024)
- [Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations](https://mlsanity.com/paper/2604.27106/) (2026)
- [Whole-body End-Effector Pose Tracking](https://mlsanity.com/paper/2409.16048/) (2024)
- [AutoURDF: Unsupervised Robot Modeling from Point Cloud Frames Using Cluster Registration](https://mlsanity.com/paper/2412.05507/) (2024)
- [Differentiable Simulation of Hard Contacts with Soft Gradients for Learning and Control](https://mlsanity.com/paper/2506.14186/) (2025)
- [3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction](https://mlsanity.com/paper/2411.00543/) (2024)
- [Scalable Non-Equivariant 3D Molecule Generation via Rotational Alignment](https://mlsanity.com/paper/2506.10186/) (2025)
- [Forecasting Continuous Non-Conservative Dynamical Systems in SO(3)](https://mlsanity.com/paper/2508.07775/) (2025)
- [Geometric Flow Matching for Molecular Conformation Generation via Manifold Decomposition](https://mlsanity.com/paper/2605.25577/) (2026)
- [MatrixNet: Learning over symmetry groups using learned group representations](https://mlsanity.com/paper/2501.09571/) (2025)
- [Continuous-Time SO(3) Forecasting with Savitzky--Golay Neural Controlled Differential Equations](https://mlsanity.com/paper/2506.06780/) (2025)
- [A Primer on SO(3) Action Representations in Deep Reinforcement Learning](https://mlsanity.com/paper/2510.11103/) (2025)
- [SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding](https://mlsanity.com/paper/2511.17411/) (2025)
- [VLD: Visual Language Goal Distance for Reinforcement Learning Navigation](https://mlsanity.com/paper/2512.07976/) (2025)
- [Reinforcement Learning with Lie Group Orientations for Robotics](https://mlsanity.com/paper/2409.11935/) (2024)
- [Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariants](https://mlsanity.com/paper/2503.15647/) (2025)
- [Learning Agile Gate Traversal via Analytical Optimal Policy Gradient](https://mlsanity.com/paper/2508.21592/) (2025)
- [Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation](https://mlsanity.com/paper/2603.19961/) (2026)
- [Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects](https://mlsanity.com/paper/2606.01950/) (2026)
- [EquiFusion: Kinematics-Agnostic Human Motion Prediction via Equivariant Latent Diffusion](https://mlsanity.com/paper/2607.10984/) (2026)
- [Cross-Attention Based Influence Model for Manual and Nonmanual Sign Language Analysis](https://mlsanity.com/paper/2409.08162/) (2024)
- [A comparison of visual representations for real-world reinforcement learning in the context of vacuum gripping](https://mlsanity.com/paper/2503.02405/) (2025)
- [FloPE: Flower Pose Estimation for Precision Pollination](https://mlsanity.com/paper/2503.11692/) (2025)
- [Rapid patient-specific neural networks for intraoperative X-ray to volume registration](https://mlsanity.com/paper/2503.16309/) (2025)

## Abstract (arXiv)

Many settings in machine learning require the selection of a rotation representation. However, choosing a suitable representation from the many available options is challenging. This paper acts as a survey and guide through rotation representations. We walk through their properties that harm or benefit deep learning with gradient-based optimization. By consolidating insights from rotation-based learning, we provide a comprehensive overview of learning functions with rotation representations. We provide guidance on selecting representations based on whether rotations are in the model's input or output and whether the data primarily comprises small angles.