# Taking a Deeper Look at the Inverse Compositional Algorithm

arXiv: https://arxiv.org/abs/1812.06861 · HTML: https://mlsanity.com/paper/1812.06861/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2018.

## Built on by (most cited first, 20 of 20)

- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [GeoCalib: Learning Single-image Calibration with Geometric Optimization](https://mlsanity.com/paper/2409.06704/) (2024)
- [ReFit: Recurrent Fitting Network for 3D Human Recovery](https://mlsanity.com/paper/2308.11184/) (2023)
- [Tangent Space Backpropagation for 3D Transformation Groups](https://mlsanity.com/paper/2103.12032/) (2021)
- [Deep Lucas-Kanade Homography for Multimodal Image Alignment](https://mlsanity.com/paper/2104.11693/) (2021)
- [Satellite Image Based Cross-view Localization for Autonomous Vehicle](https://mlsanity.com/paper/2207.13506/) (2022)
- [Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation](https://mlsanity.com/paper/2103.07153/) (2021)
- [Learning to Fit Morphable Models](https://mlsanity.com/paper/2111.14824/) (2021)
- [Residual Aligner Network](https://mlsanity.com/paper/2203.04290/) (2022)
- [VOLDOR-SLAM: For the Times When Feature-Based or Direct Methods Are Not Good Enough](https://mlsanity.com/paper/2104.06800/) (2021)
- [PINQI: An End-to-End Physics-Informed Approach to Learned Quantitative MRI Reconstruction](https://mlsanity.com/paper/2306.11023/) (2023)
- [View Birdification in the Crowd: Ground-Plane Localization from Perceived Movements](https://mlsanity.com/paper/2111.05060/) (2021)
- [ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation](https://mlsanity.com/paper/2309.14744/) (2023)
- [Reuse your features: unifying retrieval and feature-metric alignment](https://mlsanity.com/paper/2204.06292/) (2022)
- [HomoFM: Deep Homography Estimation with Flow Matching](https://mlsanity.com/paper/2601.18222/) (2026)
- [Learn2Splat: Extending the Horizon of Learned 3DGS Optimization](https://mlsanity.com/paper/2605.15760/) (2026)
- [SD-6DoF-ICLK: Sparse and Deep Inverse Compositional Lucas-Kanade Algorithm on SE(3)](https://mlsanity.com/paper/2103.16528/) (2021)
- [Probabilistic Tracking with Deep Factors](https://mlsanity.com/paper/2112.01609/) (2021)

## Abstract (arXiv)

In this paper, we provide a modern synthesis of the classic inverse compositional algorithm for dense image alignment. We first discuss the assumptions made by this well-established technique, and subsequently propose to relax these assumptions by incorporating data-driven priors into this model. More specifically, we unroll a robust version of the inverse compositional algorithm and replace multiple components of this algorithm using more expressive models whose parameters we train in an end-to-end fashion from data. Our experiments on several challenging 3D rigid motion estimation tasks demonstrate the advantages of combining optimization with learning-based techniques, outperforming the classic inverse compositional algorithm as well as data-driven image-to-pose regression approaches.