# Can representation learning for multimodal image registration be improved by supervision of intermediate layers?

arXiv: https://arxiv.org/abs/2303.00403 · HTML: https://mlsanity.com/paper/2303.00403/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 29 of 29)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [On the Representation Collapse of Sparse Mixture of Experts](https://mlsanity.com/paper/2204.09179/) (2022)
- [The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation](https://mlsanity.com/paper/1611.09326/) (2016)
- [Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2105.15134/) (2021)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations](https://mlsanity.com/paper/1903.09331/) (2019)
- [CoMIR: Contrastive Multimodal Image Representation for Registration](https://mlsanity.com/paper/2006.06325/) (2020)
- [Understanding Collapse in Non-Contrastive Siamese Representation Learning](https://mlsanity.com/paper/2209.15007/) (2022)
- [An Improved Evaluation Framework for Generative Adversarial Networks](https://mlsanity.com/paper/1803.07474/) (2018)
- [Intermediate Layers Matter in Momentum Contrastive Self Supervised Learning](https://mlsanity.com/paper/2110.14805/) (2021)
- [Toward a Geometrical Understanding of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2205.06926/) (2022)
- [Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation](https://mlsanity.com/paper/2204.13656/) (2022)
- [Is Image-to-Image Translation the Panacea for Multimodal Image Registration? A Comparative Study](https://mlsanity.com/paper/2103.16262/) (2021)
- [ContraReg: Contrastive Learning of Multi-modality Unsupervised Deformable Image Registration](https://mlsanity.com/paper/2206.13434/) (2022)
- [Fast and Robust Symmetric Image Registration Based on Distances Combining Intensity and Spatial Information](https://mlsanity.com/paper/1807.11599/) (2018)
- [TS-Net: Combining modality specific and common features for multimodal patch matching](https://mlsanity.com/paper/1806.01550/) (2018)
- [Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment](https://mlsanity.com/paper/2106.14699/) (2021)

## Built on by (most cited first, 2 of 2)

- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment](https://mlsanity.com/paper/2508.06434/) (2025)

## Abstract (arXiv)

Multimodal imaging and correlative analysis typically require image alignment. Contrastive learning can generate representations of multimodal images, reducing the challenging task of multimodal image registration to a monomodal one. Previously, additional supervision on intermediate layers in contrastive learning has improved biomedical image classification. We evaluate if a similar approach improves representations learned for registration to boost registration performance. We explore three approaches to add contrastive supervision to the latent features of the bottleneck layer in the U-Nets encoding the multimodal images and evaluate three different critic functions. Our results show that representations learned without additional supervision on latent features perform best in the downstream task of registration on two public biomedical datasets. We investigate the performance drop by exploiting recent insights in contrastive learning in classification and self-supervised learning. We visualize the spatial relations of the learned representations by means of multidimensional scaling, and show that additional supervision on the bottleneck layer can lead to partial dimensional collapse of the intermediate embedding space.