# Tangent Space Backpropagation for 3D Transformation Groups

arXiv: https://arxiv.org/abs/2103.12032 · HTML: https://mlsanity.com/paper/2103.12032/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Deep Global Registration](https://mlsanity.com/paper/2004.11540/) (2020)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)
- [A micro Lie theory for state estimation in robotics](https://mlsanity.com/paper/1812.01537/) (2018)
- [DeepFactors: Real-Time Probabilistic Dense Monocular SLAM](https://mlsanity.com/paper/2001.05049/) (2020)
- [Geoopt: Riemannian Optimization in PyTorch](https://mlsanity.com/paper/2005.02819/) (2020)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [DeepTAM: Deep Tracking and Mapping](https://mlsanity.com/paper/1808.01900/) (2018)
- [Learning multiview 3D point cloud registration](https://mlsanity.com/paper/2001.05119/) (2020)
- [Theano: A Python framework for fast computation of mathematical expressions](https://mlsanity.com/paper/1605.02688/) (2016)
- [Deep Rigid Instance Scene Flow](https://mlsanity.com/paper/1904.08913/) (2019)
- [SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks](https://mlsanity.com/paper/1606.02378/) (2016)
- [Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation](https://mlsanity.com/paper/1804.04259/) (2018)
- [gradSLAM: Automagically differentiable SLAM](https://mlsanity.com/paper/1910.10672/) (2019)
- [Taking a Deeper Look at the Inverse Compositional Algorithm](https://mlsanity.com/paper/1812.06861/) (2018)
- [McTorch, a manifold optimization library for deep learning](https://mlsanity.com/paper/1810.01811/) (2018)
- [Manifold Geometry with Fast Automatic Derivatives and Coordinate Frame Semantics Checking in C++](https://mlsanity.com/paper/1805.01810/) (2018)

## Built on by (most cited first, 30 of 40)

- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors](https://mlsanity.com/paper/2412.12392/) (2024)
- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [Approximate Differentiable Rendering with Algebraic Surfaces](https://mlsanity.com/paper/2207.10606/) (2022)
- [Flow supervision for Deformable NeRF](https://mlsanity.com/paper/2303.16333/) (2023)
- [RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimization](https://mlsanity.com/paper/2203.12870/) (2022)
- [The 8-Point Algorithm as an Inductive Bias for Relative Pose Prediction by ViTs](https://mlsanity.com/paper/2208.08988/) (2022)
- [Neural Refinement for Absolute Pose Regression with Feature Synthesis](https://mlsanity.com/paper/2303.10087/) (2023)
- [Projective Manifold Gradient Layer for Deep Rotation Regression](https://mlsanity.com/paper/2110.11657/) (2021)
- [HI-SLAM: Monocular Real-time Dense Mapping with Hybrid Implicit Fields](https://mlsanity.com/paper/2310.04787/) (2023)
- [DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark](https://mlsanity.com/paper/2403.10814/) (2024)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [M-FUSE: Multi-frame Fusion for Scene Flow Estimation](https://mlsanity.com/paper/2207.05704/) (2022)
- [D3FlowSLAM: Self-Supervised Dynamic SLAM with Flow Motion Decomposition and DINO Guidance](https://mlsanity.com/paper/2207.08794/) (2022)
- [A Laplace-inspired Distribution on SO(3) for Probabilistic Rotation Estimation](https://mlsanity.com/paper/2303.01743/) (2023)
- [CPPF++: Uncertainty-Aware Sim2Real Object Pose Estimation by Vote Aggregation](https://mlsanity.com/paper/2211.13398/) (2022)
- [PlaneRecTR++: Unified Query Learning for Joint 3D Planar Reconstruction and Pose Estimation](https://mlsanity.com/paper/2307.13756/) (2023)
- [IG-SLAM: Instant Gaussian SLAM](https://mlsanity.com/paper/2408.01126/) (2024)
- [Towards Robust Probabilistic Modeling on SO(3) via Rotation Laplace Distribution](https://mlsanity.com/paper/2305.10465/) (2023)
- [Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization](https://mlsanity.com/paper/2410.08743/) (2024)
- [MambaVO: Deep Visual Odometry Based on Sequential Matching Refinement and Training Smoothing](https://mlsanity.com/paper/2412.20082/) (2024)
- [EMR-MSF: Self-Supervised Recurrent Monocular Scene Flow Exploiting Ego-Motion Rigidity](https://mlsanity.com/paper/2309.01296/) (2023)
- [SymForce: Symbolic Computation and Code Generation for Robotics](https://mlsanity.com/paper/2204.07889/) (2022)
- [TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery](https://mlsanity.com/paper/2212.14131/) (2022)
- [Forecasting Continuous Non-Conservative Dynamical Systems in SO(3)](https://mlsanity.com/paper/2508.07775/) (2025)
- [Fit-NGP: Fitting Object Models to Neural Graphics Primitives](https://mlsanity.com/paper/2401.02357/) (2024)
- [Alignist: CAD-Informed Orientation Distribution Estimation by Fusing Shape and Correspondences](https://mlsanity.com/paper/2409.06683/) (2024)

## Abstract (arXiv)

We address the problem of performing backpropagation for computation graphs involving 3D transformation groups SO(3), SE(3), and Sim(3). 3D transformation groups are widely used in 3D vision and robotics, but they do not form vector spaces and instead lie on smooth manifolds. The standard backpropagation approach, which embeds 3D transformations in Euclidean spaces, suffers from numerical difficulties. We introduce a new library, which exploits the group structure of 3D transformations and performs backpropagation in the tangent spaces of manifolds. We show that our approach is numerically more stable, easier to implement, and beneficial to a diverse set of tasks. Our plug-and-play PyTorch library is available at https://github.com/princeton-vl/lietorch.