# MaskMedPaint: Masked Medical Image Inpainting with Diffusion Models for Mitigation of Spurious Correlations

arXiv: https://arxiv.org/abs/2411.10686 · HTML: https://mlsanity.com/paper/2411.10686/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 16 Nov 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 30)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Improving Out-of-Distribution Robustness via Selective Augmentation](https://mlsanity.com/paper/2201.00299/) (2022)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation](https://mlsanity.com/paper/2305.16289/) (2023)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [The ArtBench Dataset: Benchmarking Generative Models with Artworks](https://mlsanity.com/paper/2206.11404/) (2022)
- [Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification](https://mlsanity.com/paper/1909.01940/) (2019)
- [UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup](https://mlsanity.com/paper/2209.08928/) (2022)
- [Using Language to Extend to Unseen Domains](https://mlsanity.com/paper/2210.09520/) (2022)
- [Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases](https://mlsanity.com/paper/2212.02648/) (2022)

## Built on by (most cited first, 2 of 2)

- [AURAD: Anatomy-Pathology Unified Radiology Synthesis with Progressive Representations](https://mlsanity.com/paper/2509.04819/) (2025)
- [HistoLens: An Interactive XAI Toolkit for Verifying and Mitigating Flaws in Vision-Language Models for Histopathology](https://mlsanity.com/paper/2510.24115/) (2025)

## Abstract (arXiv)

Spurious features associated with class labels can lead image classifiers to rely on shortcuts that don't generalize well to new domains. This is especially problematic in medical settings, where biased models fail when applied to different hospitals or systems. In such cases, data-driven methods to reduce spurious correlations are preferred, as clinicians can directly validate the modified images. While Denoising Diffusion Probabilistic Models (Diffusion Models) show promise for natural images, they are impractical for medical use due to the difficulty of describing spurious medical features. To address this, we propose Masked Medical Image Inpainting (MaskMedPaint), which uses text-to-image diffusion models to augment training images by inpainting areas outside key classification regions to match the target domain. We demonstrate that MaskMedPaint enhances generalization to target domains across both natural (Waterbirds, iWildCam) and medical (ISIC 2018, Chest X-ray) datasets, given limited unlabeled target images.