mlsanityData to 7 Oct 2026

Deep Lucas-Kanade Homography for Multimodal Image Alignment

arXiv 2104.11693 ↗First posted 22 Apr 2021cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

Estimating homography to align image pairs captured by different sensors or image pairs with large appearance changes is an important and general challenge for many computer vision applications. In contrast to others, we propose a generic solution to pixel-wise align multimodal image pairs by extending the traditional Lucas-Kanade algorithm with networks. The key contribution in our method is how we construct feature maps, named as deep Lucas-Kanade feature map (DLKFM). The learned DLKFM can spontaneously recognize invariant features under various appearance-changing conditions. It also has two nice properties for the Lucas-Kanade algorithm: (1) The template feature map keeps brightness consistency with the input feature map, thus the color difference is very small while they are well-aligned. (2) The Lucas-Kanade objective function built on DLKFM has a smooth landscape around ground truth homography parameters, so the iterative solution of the Lucas-Kanade can easily converge to the ground truth. With those properties, directly updating the Lucas-Kanade algorithm on our feature maps will precisely align image pairs with large appearance changes. We share the datasets, code, and demo video online.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 21 of its 47 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

21 of 47 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Spatial Transformer Networkscited by 1,423
  3. 2019D2-Net: A Trainable CNN for Joint Detection and Description of Local Featurescited by 324
  4. 2018LF-Net: Learning Local Features from Imagescited by 151
  5. 2019GLU-Net: Global-Local Universal Network for Dense Flow and Correspondencescited by 139
  6. 2016Deep Image Homography Estimationcited by 135
  7. 2016Universal Correspondence Networkcited by 82
  8. 2017Unsupervised Deep Homography: A Fast and Robust Homography Estimation Modelcited by 75
  9. 2018HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentationcited by 66
  10. 2019Content-Aware Unsupervised Deep Homography Estimationcited by 62
  11. 2020Deep Homography Estimation for Dynamic Scenescited by 60
  12. 2018Recurrent Transformer Networks for Semantic Correspondencecited by 32
9 more
  1. 2014Do Convnets Learn Correspondence?cited by 28
  2. 2021Robust Registration of Multimodal Remote Sensing Images Based on Structural Similaritycited by 26
  3. 2019RF-Net: An End-to-End Image Matching Network based on Receptive Fieldcited by 22
  4. 2018Taking a Deeper Look at the Inverse Compositional Algorithmcited by 20
  5. 2019Semantic Attribute Matching Networkscited by 20
  6. 2019ELF: Embedded Localisation of Features in pre-trained CNNcited by 9
  7. 2017Deep-LK for Efficient Adaptive Object Trackingcited by 6
  8. 2018Fast and Robust Matching for Multimodal Remote Sensing Image Registrationcited by 6
  9. 2020Homography Estimation with Convolutional Neural Networks Under Conditions of Variancecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 6; 2024: 10; 2025: 8; 2026: 10.

38 papers.

Sort:

By area:

  1. 2023Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimatorcited by 56
  2. 2022Iterative Deep Homography Estimationcited by 45
  3. 2025MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Trainingcited by 40
  4. 2021Depth-Aware Multi-Grid Deep Homography Estimation with Contextual Correlationcited by 28
  5. 2023Deep Learning for Camera Calibration and Beyond: A Surveycited by 20
  6. 20243MOS: Multi-sources, Multi-resolutions, and Multi-scenes dataset for Optical-SAR image matchingcited by 8
  7. 2022HVC-Net: Unifying Homography, Visibility, and Confidence Learning for Planar Object Trackingcited by 7
  8. 2024SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learningcited by 7
  9. 2025Deep Learning Reforms Image Matching: A Survey and Outlookcited by 7
  10. 2021PCNet: A Structure Similarity Enhancement Method for Multispectral and Multimodal Image Registrationcited by 6
  11. 2024STHN: Deep Homography Estimation for UAV Thermal Geo-localization with Satellite Imagerycited by 5
  12. 2024Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimizationcited by 5
26 more
  1. 2025MapGlue: Multimodal Remote Sensing Image Matchingcited by 5
  2. 2023PRISE: Demystifying Deep Lucas-Kanade with Strongly Star-Convex Constraints for Multimodel Image Alignmentcited by 4
  3. 2023Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registrationcited by 4
  4. 2023Implicit Neural Image Stitchingcited by 4
  5. 2024SSHNet: Unsupervised Cross-modal Homography Estimation via Problem Reformulation and Split Optimizationcited by 4
  6. 2024Fast and Interpretable 2D Homography Decomposition: Similarity-Kernel-Similarity and Affine-Core-Affine Transformationscited by 3
  7. 2024Deep Loss Convexification for Learning Iterative Modelscited by 3
  8. 2023Panoramas from Photonscited by 2
  9. 2024VisIRNet: Deep Image Alignment for UAV-taken Visible and Infrared Image Pairscited by 2
  10. 2025OSDM-MReg: Multimodal Image Registration based One Step Diffusion Modelcited by 2
  11. 2024SeFENet: Robust Deep Homography Estimation via Semantic-Driven Feature Enhancementcited by 1
  12. 2025Auto-regressive transformation for image alignmentcited by 1
  13. 2025GLEAM: Learning to Match and Explain in Cross-View Geo-Localizationcited by 1
  14. 2026HomoFM: Deep Homography Estimation with Flow Matchingcited by 1
  15. 2026A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Datacited by 1
  16. 2026Towards Generalized Multimodal Homography Estimationcited by 1
  17. 2024Multimodal Object Detection via Probabilistic a priori Information Integrationcited by 0
  18. 2025Decoupled Geometric Parameterization and its Application in Deep Homography Estimationcited by 0
  19. 2025Collaborative Learning for Unsupervised Multimodal Remote Sensing Image Registration: Integrating Self-Supervision and MIM-Guided Diffusion-Based Image Translationcited by 0
  20. 2026HOLO: Homography-Guided Pose Estimator Network for Fine-Grained Visual Localization on SD Mapscited by 0
  21. 2026Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observationscited by 0
  22. 2026MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restorationcited by 0
  23. 2026Active Diffusion Matching: Score-based Iterative Alignment of Cross-Modal Retinal Imagescited by 0
  24. 2026Particle Diffusion Matching: Random Walk Correspondence Search for the Alignment of Standard and Ultra-Widefield Fundus Imagescited by 0
  25. 2026SA-Homo: Scale Adaptive Homography Estimation for Scale Variation Scenarioscited by 0
  26. 2026A Dynamic-Semantics Framework for Grounding Human Referring Expressions in Visual Perceptual Datacited by 0