# Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization

arXiv: https://arxiv.org/abs/2102.06315 · HTML: https://mlsanity.com/paper/2102.06315/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 11 Feb 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation](https://mlsanity.com/paper/2002.02255/) (2020)
- [Distribution Matching Losses Can Hallucinate Features in Medical Image Translation](https://mlsanity.com/paper/1805.08841/) (2018)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [Disentangled Representation Learning in Cardiac Image Analysis](https://mlsanity.com/paper/1903.09467/) (2019)
- [CycleGAN, a Master of Steganography](https://mlsanity.com/paper/1712.02950/) (2017)
- [Anatomical Priors in Convolutional Networks for Unsupervised Biomedical Segmentation](https://mlsanity.com/paper/1903.03148/) (2019)
- [Adversarial Synthesis Learning Enables Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1712.07695/) (2017)
- [Sem-GAN: Semantically-Consistent Image-to-Image Translation](https://mlsanity.com/paper/1807.04409/) (2018)
- [Adversarial Self-Defense for Cycle-Consistent GANs](https://mlsanity.com/paper/1908.01517/) (2019)
- [Harmonic Unpaired Image-to-image Translation](https://mlsanity.com/paper/1902.09727/) (2019)
- [Multi-Stage Prediction Networks for Data Harmonization](https://mlsanity.com/paper/1907.11629/) (2019)
- [INSIDE: Steering Spatial Attention with Non-Imaging Information in CNNs](https://mlsanity.com/paper/2008.10418/) (2020)
- [Using CycleGANs for effectively reducing image variability across OCT devices and improving retinal fluid segmentation](https://mlsanity.com/paper/1901.08379/) (2019)

## Built on by (most cited first, 24 of 24)

- [Multiscale Structure Guided Diffusion for Image Deblurring](https://mlsanity.com/paper/2212.01789/) (2022)
- [Domain Generalization for Medical Image Analysis: A Review](https://mlsanity.com/paper/2310.08598/) (2023)
- [Generative Adversarial Registration for Improved Conditional Deformable Templates](https://mlsanity.com/paper/2105.04349/) (2021)
- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges](https://mlsanity.com/paper/2407.00116/) (2024)
- [FedMed-GAN: Federated Domain Translation on Unsupervised Cross-Modality Brain Image Synthesis](https://mlsanity.com/paper/2201.08953/) (2022)
- [Data Harmonisation for Information Fusion in Digital Healthcare: A State-of-the-Art Systematic Review, Meta-Analysis and Future Research Directions](https://mlsanity.com/paper/2201.06505/) (2022)
- [AnyStar: Domain randomized universal star-convex 3D instance segmentation](https://mlsanity.com/paper/2307.07044/) (2023)
- [Q-space Conditioned Translation Networks for Directional Synthesis of Diffusion Weighted Images from Multi-modal Structural MRI](https://mlsanity.com/paper/2106.13188/) (2021)
- [ContraReg: Contrastive Learning of Multi-modality Unsupervised Deformable Image Registration](https://mlsanity.com/paper/2206.13434/) (2022)
- [Unsupervised Domain Adaptation with Contrastive Learning for OCT Segmentation](https://mlsanity.com/paper/2203.03664/) (2022)
- [RetiFluidNet: A Self-Adaptive and Multi-Attention Deep Convolutional Network for Retinal OCT Fluid Segmentation](https://mlsanity.com/paper/2209.12468/) (2022)
- [Medical Image Segmentation with Domain Adaptation: A Survey](https://mlsanity.com/paper/2311.01702/) (2023)
- [Pooling Image Datasets With Multiple Covariate Shift and Imbalance](https://mlsanity.com/paper/2403.02598/) (2024)
- [Joint semi-supervised and contrastive learning enables domain generalization and multi-domain segmentation](https://mlsanity.com/paper/2405.05336/) (2024)
- [Dispensed Transformer Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2110.14944/) (2021)
- [FedMed-ATL: Misaligned Unpaired Brain Image Synthesis via Affine Transform Loss](https://mlsanity.com/paper/2201.12589/) (2022)
- [TAI-GAN: A Temporally and Anatomically Informed Generative Adversarial Network for early-to-late frame conversion in dynamic cardiac PET inter-frame motion correction](https://mlsanity.com/paper/2402.09567/) (2024)
- [Diffusion based multi-domain neuroimaging harmonization method with preservation of anatomical details](https://mlsanity.com/paper/2409.00807/) (2024)
- [MRIS: A Multi-modal Retrieval Approach for Image Synthesis on Diverse Modalities](https://mlsanity.com/paper/2303.10249/) (2023)
- [MDT-Net: Multi-domain Transfer by Perceptual Supervision for Unpaired Images in OCT Scan](https://mlsanity.com/paper/2203.06363/) (2022)
- [K-Space-Aware Cross-Modality Score for Synthesized Neuroimage Quality Assessment](https://mlsanity.com/paper/2307.04296/) (2023)
- [Microscopy Image Segmentation via Point and Shape Regularized Data Synthesis](https://mlsanity.com/paper/2308.09835/) (2023)
- [Plasticine: A Traceable Diffusion Model for Medical Image Translation](https://mlsanity.com/paper/2512.18455/) (2025)

## Abstract (arXiv)

Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which limit their reliable adoption in real-world medical imaging. In this work, based on an underlying assumption that morphological shape is consistent across imaging sites, we propose a segmentation-renormalized image translation framework to reduce inter-scanner heterogeneity while preserving anatomical layout. We replace the affine transformations used in the normalization layers within generative networks with trainable scale and shift parameters conditioned on jointly learned anatomical segmentation embeddings to modulate features at every level of translation. We evaluate our methodologies against recent baselines across several imaging modalities (T1w MRI, FLAIR MRI, and OCT) on datasets with and without lesions. Segmentation-renormalization for translation GANs yields superior image harmonization as quantified by Inception distances, demonstrates improved downstream utility via post-hoc segmentation accuracy, and improved robustness to translation perturbation and self-adversarial attacks.