# Projective Manifold Gradient Layer for Deep Rotation Regression

arXiv: https://arxiv.org/abs/2110.11657 · HTML: https://mlsanity.com/paper/2110.11657/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 22 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Geometric Loss Functions for Camera Pose Regression with Deep Learning](https://mlsanity.com/paper/1704.00390/) (2017)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views](https://mlsanity.com/paper/1505.05641/) (2015)
- [An Analysis of SVD for Deep Rotation Estimation](https://mlsanity.com/paper/2006.14616/) (2020)
- [Equivariant Point Network for 3D Point Cloud Analysis](https://mlsanity.com/paper/2103.14147/) (2021)
- [Self6D: Self-Supervised Monocular 6D Object Pose Estimation](https://mlsanity.com/paper/2004.06468/) (2020)
- [Learning multiview 3D point cloud registration](https://mlsanity.com/paper/2001.05119/) (2020)
- [Deep Regression on Manifolds: A 3D Rotation Case Study](https://mlsanity.com/paper/2103.16317/) (2021)
- [Deep Part Induction from Articulated Object Pairs](https://mlsanity.com/paper/1809.07417/) (2018)
- [Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds](https://mlsanity.com/paper/1605.07147/) (2016)
- [Viewpoints and Keypoints](https://mlsanity.com/paper/1411.6067/) (2014)
- [MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization](https://mlsanity.com/paper/2101.06605/) (2021)
- [Deep Bingham Networks: Dealing with Uncertainty and Ambiguity in Pose Estimation](https://mlsanity.com/paper/2012.11002/) (2020)
- [Quaternion Equivariant Capsule Networks for 3D Point Clouds](https://mlsanity.com/paper/1912.12098/) (2019)
- [Tangent Space Backpropagation for 3D Transformation Groups](https://mlsanity.com/paper/2103.12032/) (2021)
- [Spherical Regression: Learning Viewpoints, Surface Normals and 3D Rotations on n-Spheres](https://mlsanity.com/paper/1904.05404/) (2019)
- [Deep-6DPose: Recovering 6D Object Pose from a Single RGB Image](https://mlsanity.com/paper/1802.10367/) (2018)
- [Probabilistic Permutation Synchronization using the Riemannian Structure of the Birkhoff Polytope](https://mlsanity.com/paper/1904.05814/) (2019)
- [6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference](https://mlsanity.com/paper/2004.04807/) (2020)
- [Bayesian Pose Graph Optimization via Bingham Distributions and Tempered Geodesic MCMC](https://mlsanity.com/paper/1805.12279/) (2018)
- [Robust Neural Routing Through Space Partitions for Camera Relocalization in Dynamic Indoor Environments](https://mlsanity.com/paper/2012.04746/) (2020)
- [Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry](https://mlsanity.com/paper/1805.01026/) (2018)

## Built on by (most cited first, 30 of 31)

- [RayZer: A Self-supervised Large View Synthesis Model](https://mlsanity.com/paper/2505.00702/) (2025)
- [Learning with 3D rotations, a hitchhiker's guide to SO(3)](https://mlsanity.com/paper/2404.11735/) (2024)
- [NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors](https://mlsanity.com/paper/2403.03122/) (2024)
- [GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion](https://mlsanity.com/paper/2309.15459/) (2023)
- [FisherMatch: Semi-Supervised Rotation Regression via Entropy-based Filtering](https://mlsanity.com/paper/2203.15765/) (2022)
- [Non-Rigid Shape Registration via Deep Functional Maps Prior](https://mlsanity.com/paper/2311.04494/) (2023)
- [SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation](https://mlsanity.com/paper/2409.11682/) (2024)
- [A Laplace-inspired Distribution on SO(3) for Probabilistic Rotation Estimation](https://mlsanity.com/paper/2303.01743/) (2023)
- [Learning Generalizable Tool-use Skills through Trajectory Generation](https://mlsanity.com/paper/2310.00156/) (2023)
- [UP-SLAM: Adaptively Structured Gaussian SLAM with Uncertainty Prediction in Dynamic Environments](https://mlsanity.com/paper/2505.22335/) (2025)
- [Geometric Neural Distance Fields for Learning Human Motion Priors](https://mlsanity.com/paper/2509.09667/) (2025)
- [Towards Robust Probabilistic Modeling on SO(3) via Rotation Laplace Distribution](https://mlsanity.com/paper/2305.10465/) (2023)
- [3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction](https://mlsanity.com/paper/2411.00543/) (2024)
- [Learning to Localize in Unseen Scenes with Relative Pose Regressors](https://mlsanity.com/paper/2303.02717/) (2023)
- [Causal Transformer for Fusion and Pose Estimation in Deep Visual Inertial Odometry](https://mlsanity.com/paper/2409.08769/) (2024)
- [Forecasting Continuous Non-Conservative Dynamical Systems in SO(3)](https://mlsanity.com/paper/2508.07775/) (2025)
- [Category-Level Object Shape and Pose Estimation in Less Than a Millisecond](https://mlsanity.com/paper/2509.18979/) (2025)
- [TetraSphere: A Neural Descriptor for O(3)-Invariant Point Cloud Analysis](https://mlsanity.com/paper/2211.14456/) (2022)
- [Alignist: CAD-Informed Orientation Distribution Estimation by Fusing Shape and Correspondences](https://mlsanity.com/paper/2409.06683/) (2024)
- [Continuous-Time SO(3) Forecasting with Savitzky--Golay Neural Controlled Differential Equations](https://mlsanity.com/paper/2506.06780/) (2025)
- [Parallelised Differentiable Straightest Geodesics for 3D Meshes](https://mlsanity.com/paper/2603.15780/) (2026)
- [Deep Projective Rotation Estimation through Relative Supervision](https://mlsanity.com/paper/2211.11182/) (2022)
- [A survey of manifold learning and its applications for multimedia](https://mlsanity.com/paper/2310.12986/) (2023)
- [PDF-HR: Pose Distance Fields for Humanoid Robots](https://mlsanity.com/paper/2602.04851/) (2026)
- [Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation](https://mlsanity.com/paper/2603.19961/) (2026)
- [Changes from Classical Statistics to Modern Statistics and Data Science](https://mlsanity.com/paper/2211.03756/) (2022)
- [Deep Extrinsic Manifold Representation for Vision Tasks](https://mlsanity.com/paper/2404.00544/) (2024)
- [NFR: Neural Feature-Guided Non-Rigid Shape Registration](https://mlsanity.com/paper/2505.22445/) (2025)
- [RA-NeRF: Robust Neural Radiance Field Reconstruction with Accurate Camera Pose Estimation under Complex Trajectories](https://mlsanity.com/paper/2506.15242/) (2025)
- [DAP-Pose: Deep Temporal Alignment and Physics-aware Cross-modal Sensor Fusion for Robust Pose Estimation](https://mlsanity.com/paper/2607.23755/) (2026)

## Abstract (arXiv)

Regressing rotations on SO(3) manifold using deep neural networks is an important yet unsolved problem. The gap between the Euclidean network output space and the non-Euclidean SO(3) manifold imposes a severe challenge for neural network learning in both forward and backward passes. While several works have proposed different regression-friendly rotation representations, very few works have been devoted to improving the gradient backpropagating in the backward pass. In this paper, we propose a manifold-aware gradient that directly backpropagates into deep network weights. Leveraging Riemannian optimization to construct a novel projective gradient, our proposed regularized projective manifold gradient (RPMG) method helps networks achieve new state-of-the-art performance in a variety of rotation estimation tasks. Our proposed gradient layer can also be applied to other smooth manifolds such as the unit sphere. Our project page is at https://jychen18.github.io/RPMG.