# CF-Seg: Counterfactuals meet Segmentation

arXiv: https://arxiv.org/abs/2506.16213 · HTML: https://mlsanity.com/paper/2506.16213/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 22 of 22)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training](https://mlsanity.com/paper/1709.02023/) (2017)
- [Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation](https://mlsanity.com/paper/1705.08302/) (2017)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [Neural Causal Models for Counterfactual Identification and Estimation](https://mlsanity.com/paper/2210.00035/) (2022)
- [High Fidelity Image Counterfactuals with Probabilistic Causal Models](https://mlsanity.com/paper/2306.15764/) (2023)
- [Post-DAE: Anatomically Plausible Segmentation via Post-Processing with Denoising Autoencoders](https://mlsanity.com/paper/2006.13791/) (2020)
- [Equitable modelling of brain imaging by counterfactual augmentation with morphologically constrained 3D deep generative models](https://mlsanity.com/paper/2111.14923/) (2021)
- [Enhancing Spatiotemporal Disease Progression Models via Latent Diffusion and Prior Knowledge](https://mlsanity.com/paper/2405.03328/) (2024)
- [CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images](https://mlsanity.com/paper/2307.03293/) (2023)
- [Pseudo-healthy synthesis with pathology disentanglement and adversarial learning](https://mlsanity.com/paper/2005.01607/) (2020)
- [Shape-Aware Complementary-Task Learning for Multi-Organ Segmentation](https://mlsanity.com/paper/1908.05099/) (2019)
- [Many tasks make light work: Learning to localise medical anomalies from multiple synthetic tasks](https://mlsanity.com/paper/2307.00899/) (2023)
- [Debiasing Counterfactuals In the Presence of Spurious Correlations](https://mlsanity.com/paper/2308.10984/) (2023)
- [Counterfactual contrastive learning: robust representations via causal image synthesis](https://mlsanity.com/paper/2403.09605/) (2024)
- [Anatomy-Driven Pathology Detection on Chest X-rays](https://mlsanity.com/paper/2309.02578/) (2023)
- [Evaluation of pseudo-healthy image reconstruction for anomaly detection with deep generative models: Application to brain FDG PET](https://mlsanity.com/paper/2401.16363/) (2024)
- [Anatomy-guided Pathology Segmentation](https://mlsanity.com/paper/2407.05844/) (2024)
- [DeCoDEx: Confounder Detector Guidance for Improved Diffusion-based Counterfactual Explanations](https://mlsanity.com/paper/2405.09288/) (2024)
- [Implicit Counterfactual Data Augmentation for Robust Learning](https://mlsanity.com/paper/2304.13431/) (2023)

## Built on by (most cited first, 10 of 10)

- [Counterfactual Identifiability via Dynamic Optimal Transport](https://mlsanity.com/paper/2510.08294/) (2025)
- [Flow Stochastic Segmentation Networks](https://mlsanity.com/paper/2507.18838/) (2025)
- [CheXmask-U: Quantifying uncertainty in landmark-based anatomical segmentation for X-ray images](https://mlsanity.com/paper/2512.10715/) (2025)
- [Scaling Generative Foundation Models for Chest Radiography with Rectified Flow Transformers](https://mlsanity.com/paper/2606.19460/) (2026)
- [Pixel-level Counterfactual Contrastive Learning for Medical Image Segmentation](https://mlsanity.com/paper/2603.17110/) (2026)
- [Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Models](https://mlsanity.com/paper/2609.24879/) (2026)
- [Hide-and-Seek Attribution: Weakly Supervised Segmentation of Vertebral Metastases in CT](https://mlsanity.com/paper/2512.06849/) (2025)
- [Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis](https://mlsanity.com/paper/2603.21213/) (2026)
- [Counterfactual Stress Testing for Image Classification Models](https://mlsanity.com/paper/2605.10894/) (2026)
- [TRACE-Seg3D: Counterfactual Context Auditing For Robust 3D Glioma Segmentation Under Institutional Shift](https://mlsanity.com/paper/2607.07038/) (2026)

## Abstract (arXiv)

Segmenting anatomical structures in medical images plays an important role in the quantitative assessment of various diseases. However, accurate segmentation becomes significantly more challenging in the presence of disease. Disease patterns can alter the appearance of surrounding healthy tissues, introduce ambiguous boundaries, or even obscure critical anatomical structures. As such, segmentation models trained on real-world datasets may struggle to provide good anatomical segmentation, leading to potential misdiagnosis. In this paper, we generate counterfactual (CF) images to simulate how the same anatomy would appear in the absence of disease without altering the underlying structure. We then use these CF images to segment structures of interest, without requiring any changes to the underlying segmentation model. Our experiments on two real-world clinical chest X-ray datasets show that the use of counterfactual images improves anatomical segmentation, thereby aiding downstream clinical decision-making.