# Vista-Morph: Unsupervised Image Registration of Visible-Thermal Facial Pairs

arXiv: https://arxiv.org/abs/2306.06505 · HTML: https://mlsanity.com/paper/2306.06505/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 55 references that are in the map.
- First posted on arXiv 10 Jun 2023.

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [The relativistic discriminator: a key element missing from standard GAN](https://mlsanity.com/paper/1807.00734/) (2018)
- [VT-ADL: A Vision Transformer Network for Image Anomaly Detection and Localization](https://mlsanity.com/paper/2104.10036/) (2021)
- [A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration](https://mlsanity.com/paper/1809.06130/) (2018)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration](https://mlsanity.com/paper/2205.11876/) (2022)
- [Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://mlsanity.com/paper/1907.06724/) (2019)
- [Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation](https://mlsanity.com/paper/2003.08073/) (2020)
- [A Large-Scale, Time-Synchronized Visible and Thermal Face Dataset](https://mlsanity.com/paper/2101.02637/) (2021)
- [T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2209.08814/) (2022)
- [TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition](https://mlsanity.com/paper/1712.02514/) (2017)
- [Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation](https://mlsanity.com/paper/2204.13656/) (2022)
- [A new face database simultaneously acquired in visible, near infrared and thermal spectrum](https://mlsanity.com/paper/2202.13864/) (2022)
- [Matching Thermal to Visible Face Images Using a Semantic-Guided Generative Adversarial Network](https://mlsanity.com/paper/1903.00963/) (2019)
- [The Use of AI for Thermal Emotion Recognition: A Review of Problems and Limitations in Standard Design and Data](https://mlsanity.com/paper/2009.10589/) (2020)
- [When Visible-to-Thermal Facial GAN Beats Conditional Diffusion](https://mlsanity.com/paper/2302.09395/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Generative Approach for Image Registration of Visible-Thermal (VT) Cancer Faces](https://mlsanity.com/paper/2308.12271/) (2023)

## Abstract (arXiv)

For a variety of biometric cross-spectral tasks, Visible-Thermal (VT) facial pairs are used. However, due to a lack of calibration in the lab, photographic capture between two different sensors leads to severely misaligned pairs that can lead to poor results for person re-identification and generative AI. To solve this problem, we introduce our approach for VT image registration called Vista Morph. Unlike existing VT facial registration that requires manual, hand-crafted features for pixel matching and/or a supervised thermal reference, Vista Morph is completely unsupervised without the need for a reference. By learning the affine matrix through a Vision Transformer (ViT)-based Spatial Transformer Network (STN) and Generative Adversarial Networks (GAN), Vista Morph successfully aligns facial and non-facial VT images. Our approach learns warps in Hard, No, and Low-light visual settings and is robust to geometric perturbations and erasure at test time. We conduct a downstream generative AI task to show that registering training data with Vista Morph improves subject identity of generated thermal faces when performing V2T image translation.