# The Devil is in the Prompts: De-Identification Traces Enhance Memorization Risks in Synthetic Chest X-Ray Generation

arXiv: https://arxiv.org/abs/2502.07516 · HTML: https://mlsanity.com/paper/2502.07516/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 20 of 20)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Understanding and Mitigating Copying in Diffusion Models](https://mlsanity.com/paper/2305.20086/) (2023)
- [Detecting, Explaining, and Mitigating Memorization in Diffusion Models](https://mlsanity.com/paper/2407.21720/) (2024)
- [Generative models improve fairness of medical classifiers under distribution shifts](https://mlsanity.com/paper/2304.09218/) (2023)
- [Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention](https://mlsanity.com/paper/2403.11052/) (2024)
- [Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity](https://mlsanity.com/paper/2305.08252/) (2023)
- [On the De-duplication of LAION-2B](https://mlsanity.com/paper/2303.12733/) (2023)
- [RadEdit: stress-testing biomedical vision models via diffusion image editing](https://mlsanity.com/paper/2312.12865/) (2023)
- [Beware of diffusion models for synthesizing medical images -- A comparison with GANs in terms of memorizing brain MRI and chest x-ray images](https://mlsanity.com/paper/2305.07644/) (2023)
- [Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis](https://mlsanity.com/paper/2307.01148/) (2023)
- [Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models](https://mlsanity.com/paper/2306.01322/) (2023)
- [Using Diffusion Models to Generate Synthetic Labelled Data for Medical Image Segmentation](https://mlsanity.com/paper/2310.16794/) (2023)
- [From Majority to Minority: A Diffusion-based Augmentation for Underrepresented Groups in Skin Lesion Analysis](https://mlsanity.com/paper/2406.18375/) (2024)
- [Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models](https://mlsanity.com/paper/2410.22149/) (2024)

## Built on by (most cited first, 3 of 3)

- [CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs](https://mlsanity.com/paper/2505.10496/) (2025)
- [Diffusion Models Memorize in Training -- and Generalize in Inference](https://mlsanity.com/paper/2603.13419/) (2026)
- [A Utility-preserving De-identification Pipeline for Cross-hospital Radiology Data Sharing](https://mlsanity.com/paper/2604.07128/) (2026)

## Abstract (arXiv)

Generative models, particularly text-to-image (T2I) diffusion models, play a crucial role in medical image analysis. However, these models are prone to training data memorization, posing significant risks to patient privacy. Synthetic chest X-ray generation is one of the most common applications in medical image analysis with the MIMIC-CXR dataset serving as the primary data repository for this task. This study presents the first systematic attempt to identify prompts and text tokens in MIMIC-CXR that contribute the most to training data memorization. Our analysis reveals two unexpected findings: (1) prompts containing traces of de-identification procedures (markers introduced to hide Protected Health Information) are the most memorized, and (2) among all tokens, de-identification markers contribute the most towards memorization. This highlights a broader issue with the standard anonymization practices and T2I synthesis with MIMIC-CXR. To exacerbate, existing inference-time memorization mitigation strategies are ineffective and fail to sufficiently reduce the model's reliance on memorized text tokens. On this front, we propose actionable strategies for different stakeholders to enhance privacy and improve the reliability of generative models in medical imaging. Finally, our results provide a foundation for future work on developing and benchmarking memorization mitigation techniques for synthetic chest X-ray generation using the MIMIC-CXR dataset. The anonymized code is available at https://anonymous.4open.science/r/diffusion_memorization-8011/