# Inverse Compositional Spatial Transformer Networks

arXiv: https://arxiv.org/abs/1612.03897 · HTML: https://mlsanity.com/paper/1612.03897/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2016.

## Built on by (most cited first, 29 of 29)

- [DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video](https://mlsanity.com/paper/2303.03988/) (2023)
- [GAN-Supervised Dense Visual Alignment](https://mlsanity.com/paper/2112.05143/) (2021)
- [Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification](https://mlsanity.com/paper/2101.06635/) (2021)
- [Scale-invariant scale-channel networks: Deep networks that generalise to previously unseen scales](https://mlsanity.com/paper/2106.06418/) (2021)
- [Deep Feedback Inverse Problem Solver](https://mlsanity.com/paper/2101.07719/) (2021)
- [Canonical Factors for Hybrid Neural Fields](https://mlsanity.com/paper/2308.15461/) (2023)
- [Learning Canonical 3D Object Representation for Fine-Grained Recognition](https://mlsanity.com/paper/2108.04628/) (2021)
- [HVC-Net: Unifying Homography, Visibility, and Confidence Learning for Planar Object Tracking](https://mlsanity.com/paper/2209.08924/) (2022)
- [Learning to See Through with Events](https://mlsanity.com/paper/2212.02219/) (2022)
- [Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers](https://mlsanity.com/paper/2405.03730/) (2024)
- [SpaceJAM: a Lightweight and Regularization-free Method for Fast Joint Alignment of Images](https://mlsanity.com/paper/2407.11850/) (2024)
- [Adjoint Rigid Transform Network: Task-conditioned Alignment of 3D Shapes](https://mlsanity.com/paper/2102.01161/) (2021)
- [CNN-based Ego-Motion Estimation for Fast MAV Maneuvers](https://mlsanity.com/paper/2101.01841/) (2021)
- [High-Quality 3D Face Reconstruction with Affine Convolutional Networks](https://mlsanity.com/paper/2310.14237/) (2023)
- [Metric Convolutions: A Unifying Theory to Adaptive Image Convolutions](https://mlsanity.com/paper/2406.05400/) (2024)
- [Diffeomorphic Temporal Alignment Nets for Time-series Joint Alignment and Averaging](https://mlsanity.com/paper/2502.06591/) (2025)
- [Geometrically Constrained and Token-Based Probabilistic Spatial Transformers](https://mlsanity.com/paper/2509.11218/) (2025)
- [The Devils in the Point Clouds: Studying the Robustness of Point Cloud Convolutions](https://mlsanity.com/paper/2101.07832/) (2021)
- [Spatial Transformation for Image Composition via Correspondence Learning](https://mlsanity.com/paper/2207.02398/) (2022)
- [TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation](https://mlsanity.com/paper/2303.13477/) (2023)
- [Saccadic Vision for Fine-Grained Visual Classification](https://mlsanity.com/paper/2509.15688/) (2025)
- [Human Pose Transfer with Augmented Disentangled Feature Consistency](https://mlsanity.com/paper/2107.10984/) (2021)
- [Perspective Transformation Layer](https://mlsanity.com/paper/2201.05706/) (2022)
- [Spatial-Temporal Alignment Network for Action Recognition](https://mlsanity.com/paper/2308.09897/) (2023)
- [GeoDeformer: Geometric Deformable Transformer for Action Recognition](https://mlsanity.com/paper/2311.17975/) (2023)
- [Adaptive Low Light Enhancement via Joint Global-Local Illumination Adjustment](https://mlsanity.com/paper/2504.00400/) (2025)
- [Robust Canonicalization through Bootstrapped Data Re-Alignment](https://mlsanity.com/paper/2510.08178/) (2025)
- [FastJAM: a Fast Joint Alignment Model for Images](https://mlsanity.com/paper/2510.22842/) (2025)
- [Entropy Transformer Networks: A Learning Approach via Tangent Bundle Data Manifold](https://mlsanity.com/paper/2307.12517/) (2023)

## Abstract (arXiv)

In this paper, we establish a theoretical connection between the classical Lucas & Kanade (LK) algorithm and the emerging topic of Spatial Transformer Networks (STNs). STNs are of interest to the vision and learning communities due to their natural ability to combine alignment and classification within the same theoretical framework. Inspired by the Inverse Compositional (IC) variant of the LK algorithm, we present Inverse Compositional Spatial Transformer Networks (IC-STNs). We demonstrate that IC-STNs can achieve better performance than conventional STNs with less model capacity; in particular, we show superior performance in pure image alignment tasks as well as joint alignment/classification problems on real-world problems.