# Deep Lucas-Kanade Homography for Multimodal Image Alignment

arXiv: https://arxiv.org/abs/2104.11693 · HTML: https://mlsanity.com/paper/2104.11693/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 22 Apr 2021.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [Deep Image Homography Estimation](https://mlsanity.com/paper/1606.03798/) (2016)
- [Universal Correspondence Network](https://mlsanity.com/paper/1606.03558/) (2016)
- [Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model](https://mlsanity.com/paper/1709.03966/) (2017)
- [HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation](https://mlsanity.com/paper/1804.02967/) (2018)
- [Content-Aware Unsupervised Deep Homography Estimation](https://mlsanity.com/paper/1909.05983/) (2019)
- [Deep Homography Estimation for Dynamic Scenes](https://mlsanity.com/paper/2004.02132/) (2020)
- [Recurrent Transformer Networks for Semantic Correspondence](https://mlsanity.com/paper/1810.12155/) (2018)
- [Do Convnets Learn Correspondence?](https://mlsanity.com/paper/1411.1091/) (2014)
- [Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity](https://mlsanity.com/paper/2103.16871/) (2021)
- [RF-Net: An End-to-End Image Matching Network based on Receptive Field](https://mlsanity.com/paper/1906.00604/) (2019)
- [Taking a Deeper Look at the Inverse Compositional Algorithm](https://mlsanity.com/paper/1812.06861/) (2018)
- [Semantic Attribute Matching Networks](https://mlsanity.com/paper/1904.02969/) (2019)
- [ELF: Embedded Localisation of Features in pre-trained CNN](https://mlsanity.com/paper/1907.03261/) (2019)
- [Deep-LK for Efficient Adaptive Object Tracking](https://mlsanity.com/paper/1705.06839/) (2017)
- [Fast and Robust Matching for Multimodal Remote Sensing Image Registration](https://mlsanity.com/paper/1808.06194/) (2018)
- [Homography Estimation with Convolutional Neural Networks Under Conditions of Variance](https://mlsanity.com/paper/2010.01041/) (2020)

## Built on by (most cited first, 30 of 38)

- [Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator](https://mlsanity.com/paper/2308.16906/) (2023)
- [Iterative Deep Homography Estimation](https://mlsanity.com/paper/2203.15982/) (2022)
- [MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training](https://mlsanity.com/paper/2501.07556/) (2025)
- [Depth-Aware Multi-Grid Deep Homography Estimation with Contextual Correlation](https://mlsanity.com/paper/2107.02524/) (2021)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [3MOS: Multi-sources, Multi-resolutions, and Multi-scenes dataset for Optical-SAR image matching](https://mlsanity.com/paper/2404.00838/) (2024)
- [HVC-Net: Unifying Homography, Visibility, and Confidence Learning for Planar Object Tracking](https://mlsanity.com/paper/2209.08924/) (2022)
- [STHN: Deep Homography Estimation for UAV Thermal Geo-localization with Satellite Imagery](https://mlsanity.com/paper/2405.20470/) (2024)
- [SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning](https://mlsanity.com/paper/2407.08148/) (2024)
- [Deep Learning Reforms Image Matching: A Survey and Outlook](https://mlsanity.com/paper/2506.04619/) (2025)
- [PCNet: A Structure Similarity Enhancement Method for Multispectral and Multimodal Image Registration](https://mlsanity.com/paper/2106.05124/) (2021)
- [Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization](https://mlsanity.com/paper/2411.13036/) (2024)
- [MapGlue: Multimodal Remote Sensing Image Matching](https://mlsanity.com/paper/2503.16185/) (2025)
- [PRISE: Demystifying Deep Lucas-Kanade with Strongly Star-Convex Constraints for Multimodel Image Alignment](https://mlsanity.com/paper/2303.11526/) (2023)
- [Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registration](https://mlsanity.com/paper/2304.05646/) (2023)
- [Implicit Neural Image Stitching](https://mlsanity.com/paper/2309.01409/) (2023)
- [SSHNet: Unsupervised Cross-modal Homography Estimation via Problem Reformulation and Split Optimization](https://mlsanity.com/paper/2409.17993/) (2024)
- [Fast and Interpretable 2D Homography Decomposition: Similarity-Kernel-Similarity and Affine-Core-Affine Transformations](https://mlsanity.com/paper/2402.18008/) (2024)
- [Deep Loss Convexification for Learning Iterative Models](https://mlsanity.com/paper/2411.10649/) (2024)
- [Panoramas from Photons](https://mlsanity.com/paper/2309.03811/) (2023)
- [VisIRNet: Deep Image Alignment for UAV-taken Visible and Infrared Image Pairs](https://mlsanity.com/paper/2402.09635/) (2024)
- [OSDM-MReg: Multimodal Image Registration based One Step Diffusion Model](https://mlsanity.com/paper/2504.06027/) (2025)
- [SeFENet: Robust Deep Homography Estimation via Semantic-Driven Feature Enhancement](https://mlsanity.com/paper/2412.06352/) (2024)
- [Auto-regressive transformation for image alignment](https://mlsanity.com/paper/2505.04864/) (2025)
- [GLEAM: Learning to Match and Explain in Cross-View Geo-Localization](https://mlsanity.com/paper/2509.07450/) (2025)
- [HomoFM: Deep Homography Estimation with Flow Matching](https://mlsanity.com/paper/2601.18222/) (2026)
- [A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Data](https://mlsanity.com/paper/2602.19562/) (2026)
- [Towards Generalized Multimodal Homography Estimation](https://mlsanity.com/paper/2603.03956/) (2026)
- [Multimodal Object Detection via Probabilistic a priori Information Integration](https://mlsanity.com/paper/2405.15596/) (2024)
- [Decoupled Geometric Parameterization and its Application in Deep Homography Estimation](https://mlsanity.com/paper/2505.16599/) (2025)

## Abstract (arXiv)

Estimating homography to align image pairs captured by different sensors or image pairs with large appearance changes is an important and general challenge for many computer vision applications. In contrast to others, we propose a generic solution to pixel-wise align multimodal image pairs by extending the traditional Lucas-Kanade algorithm with networks. The key contribution in our method is how we construct feature maps, named as deep Lucas-Kanade feature map (DLKFM). The learned DLKFM can spontaneously recognize invariant features under various appearance-changing conditions. It also has two nice properties for the Lucas-Kanade algorithm: (1) The template feature map keeps brightness consistency with the input feature map, thus the color difference is very small while they are well-aligned. (2) The Lucas-Kanade objective function built on DLKFM has a smooth landscape around ground truth homography parameters, so the iterative solution of the Lucas-Kanade can easily converge to the ground truth. With those properties, directly updating the Lucas-Kanade algorithm on our feature maps will precisely align image pairs with large appearance changes. We share the datasets, code, and demo video online.