# Deep Regression on Manifolds: A 3D Rotation Case Study

arXiv: https://arxiv.org/abs/2103.16317 · HTML: https://mlsanity.com/paper/2103.16317/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [BOP Challenge 2020 on 6D Object Localization](https://mlsanity.com/paper/2009.07378/) (2020)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [An Analysis of SVD for Deep Rotation Estimation](https://mlsanity.com/paper/2006.14616/) (2020)
- [TexturePose: Supervising Human Mesh Estimation with Texture Consistency](https://mlsanity.com/paper/1910.11322/) (2019)
- [On Pre-Trained Image Features and Synthetic Images for Deep Learning](https://mlsanity.com/paper/1710.10710/) (2017)
- [Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wild](https://mlsanity.com/paper/1908.06442/) (2019)
- [Spherical Regression: Learning Viewpoints, Surface Normals and 3D Rotations on n-Spheres](https://mlsanity.com/paper/1904.05404/) (2019)
- [Probabilistic orientation estimation with matrix Fisher distributions](https://mlsanity.com/paper/2006.09740/) (2020)
- [On Object Symmetries and 6D Pose Estimation from Images](https://mlsanity.com/paper/1908.07640/) (2019)
- [A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty](https://mlsanity.com/paper/2006.01031/) (2020)
- [Defining the Pose of any 3D Rigid Object and an Associated Distance](https://mlsanity.com/paper/1612.04631/) (2016)
- [A compact formula for the derivative of a 3-D rotation in exponential coordinates](https://mlsanity.com/paper/1312.0788/) (2013)

## Built on by (most cited first, 30 of 67)

- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality](https://mlsanity.com/paper/2210.13702/) (2022)
- [Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot](https://mlsanity.com/paper/2402.14654/) (2024)
- [PoseGPT: Quantization-based 3D Human Motion Generation and Forecasting](https://mlsanity.com/paper/2210.10542/) (2022)
- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [Scene Coordinate Reconstruction: Posing of Image Collections via Incremental Learning of a Relocalizer](https://mlsanity.com/paper/2404.14351/) (2024)
- [Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors](https://mlsanity.com/paper/2503.17316/) (2025)
- [Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching](https://mlsanity.com/paper/2409.07343/) (2024)
- [CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects](https://mlsanity.com/paper/2303.15782/) (2023)
- [LoopSplat: Loop Closure by Registering 3D Gaussian Splats](https://mlsanity.com/paper/2408.10154/) (2024)
- [Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2211.11505/) (2022)
- [gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction](https://mlsanity.com/paper/2304.11970/) (2023)
- [Learning with 3D rotations, a hitchhiker's guide to SO(3)](https://mlsanity.com/paper/2404.11735/) (2024)
- [Projective Manifold Gradient Layer for Deep Rotation Regression](https://mlsanity.com/paper/2110.11657/) (2021)
- [Scaling 4D Representations](https://mlsanity.com/paper/2412.15212/) (2024)
- [RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination](https://mlsanity.com/paper/2505.21925/) (2025)
- [SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments](https://mlsanity.com/paper/2303.09555/) (2023)
- [Image to Sphere: Learning Equivariant Features for Efficient Pose Prediction](https://mlsanity.com/paper/2302.13926/) (2023)
- [SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstruction](https://mlsanity.com/paper/2309.10748/) (2023)
- [End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon](https://mlsanity.com/paper/2309.16634/) (2023)
- [PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling](https://mlsanity.com/paper/2208.10211/) (2022)
- [SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars](https://mlsanity.com/paper/2410.11682/) (2024)
- [From Image to Video: An Empirical Study of Diffusion Representations](https://mlsanity.com/paper/2502.07001/) (2025)
- [Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrases](https://mlsanity.com/paper/2310.04189/) (2023)
- [CenterGrasp: Object-Aware Implicit Representation Learning for Simultaneous Shape Reconstruction and 6-DoF Grasp Estimation](https://mlsanity.com/paper/2312.08240/) (2023)
- [Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image](https://mlsanity.com/paper/2504.03177/) (2025)
- [Spacecraft Pose Estimation Based on Unsupervised Domain Adaptation and on a 3D-Guided Loss Combination](https://mlsanity.com/paper/2212.13415/) (2022)
- [MFOS: Model-Free & One-Shot Object Pose Estimation](https://mlsanity.com/paper/2310.01897/) (2023)
- [Flexible MOF Generation with Torsion-Aware Flow Matching](https://mlsanity.com/paper/2505.17914/) (2025)
- [Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision](https://mlsanity.com/paper/2501.01715/) (2025)

## Abstract (arXiv)

Many machine learning problems involve regressing variables on a non-Euclidean manifold -- e.g. a discrete probability distribution, or the 6D pose of an object. One way to tackle these problems through gradient-based learning is to use a differentiable function that maps arbitrary inputs of a Euclidean space onto the manifold. In this paper, we establish a set of desirable properties for such mapping, and in particular highlight the importance of pre-images connectivity/convexity. We illustrate these properties with a case study regarding 3D rotations. Through theoretical considerations and methodological experiments on a variety of tasks, we review various differentiable mappings on the 3D rotation space, and conjecture about the importance of their local linearity. We show that a mapping based on Procrustes orthonormalization generally performs best among the mappings considered, but that a rotation vector representation might also be suitable when restricted to small angles.