# Equitable modelling of brain imaging by counterfactual augmentation with morphologically constrained 3D deep generative models

arXiv: https://arxiv.org/abs/2111.14923 · HTML: https://mlsanity.com/paper/2111.14923/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 117 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [Fairness in Machine Learning](https://mlsanity.com/paper/2012.15816/) (2020)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study](https://mlsanity.com/paper/2004.03271/) (2020)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [Brain Tumor Segmentation with Deep Neural Networks](https://mlsanity.com/paper/1505.03540/) (2015)
- [Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration](https://mlsanity.com/paper/1805.04605/) (2018)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)

## Built on by (most cited first, 30 of 36)

- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation](https://mlsanity.com/paper/2312.14223/) (2023)
- [Enhancing Spatiotemporal Disease Progression Models via Latent Diffusion and Prior Knowledge](https://mlsanity.com/paper/2405.03328/) (2024)
- [Addressing Fairness Issues in Deep Learning-Based Medical Image Analysis: A Systematic Review](https://mlsanity.com/paper/2209.13177/) (2022)
- [TADM: Temporally-Aware Diffusion Model for Neurodegenerative Progression on Brain MRI](https://mlsanity.com/paper/2406.12411/) (2024)
- [Brain Latent Progression: Individual-based Spatiotemporal Disease Progression on 3D Brain MRIs via Latent Diffusion](https://mlsanity.com/paper/2502.08560/) (2025)
- [A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Science](https://mlsanity.com/paper/2403.12636/) (2024)
- [Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis](https://mlsanity.com/paper/2412.20651/) (2024)
- [CF-Seg: Counterfactuals meet Segmentation](https://mlsanity.com/paper/2506.16213/) (2025)
- [Synthesizing Individualized Aging Brains in Health and Disease with Generative Models and Parallel Transport](https://mlsanity.com/paper/2502.21049/) (2025)
- [Multiscale Metamorphic VAE for 3D Brain MRI Synthesis](https://mlsanity.com/paper/2301.03588/) (2023)
- [Learning Patient-Specific Disease Dynamics with Latent Flow Matching for Longitudinal Imaging Generation](https://mlsanity.com/paper/2512.09185/) (2025)
- [FlowLet: Conditional 3D Brain MRI Synthesis using Wavelet Flow Matching](https://mlsanity.com/paper/2601.05212/) (2026)
- [3D Brain and Heart Volume Generative Models: A Survey](https://mlsanity.com/paper/2210.05952/) (2022)
- [A Unified Framework for Generative Data Augmentation: A Comprehensive Survey](https://mlsanity.com/paper/2310.00277/) (2023)
- [Synthetic Data for Robust Stroke Segmentation](https://mlsanity.com/paper/2404.01946/) (2024)
- [Deep Generative Classification of Blood Cell Morphology](https://mlsanity.com/paper/2408.08982/) (2024)
- [Representational Ethical Model Calibration](https://mlsanity.com/paper/2207.12043/) (2022)
- [IdenBAT: Disentangled Representation Learning for Identity-Preserved Brain Age Transformation](https://mlsanity.com/paper/2410.16945/) (2024)
- [Temporally-Aware Diffusion Model for Brain Progression Modelling with Bidirectional Temporal Regularisation](https://mlsanity.com/paper/2509.03141/) (2025)
- [Neural Autoregressive Modeling of Brain Aging](https://mlsanity.com/paper/2507.22954/) (2025)
- [Anatomically Guided Latent Diffusion for Brain MRI Progression Modeling](https://mlsanity.com/paper/2601.14584/) (2026)
- [Deep Variational Lesion-Deficit Mapping](https://mlsanity.com/paper/2305.17478/) (2023)
- [A Quantitatively Interpretable Model for Alzheimer's Disease Prediction Using Deep Counterfactuals](https://mlsanity.com/paper/2310.03457/) (2023)
- [MambaControl: Anatomy Graph-Enhanced Mamba ControlNet with Fourier Refinement for Diffusion-Based Disease Trajectory Prediction](https://mlsanity.com/paper/2505.09965/) (2025)
- [Integrating Anatomical Priors into a Causal Diffusion Model](https://mlsanity.com/paper/2509.09054/) (2025)
- [Cycle Diffusion Model for Counterfactual Image Generation](https://mlsanity.com/paper/2509.24267/) (2025)
- [Forecasting Future Anatomies: Longitudinal Brain Mri-to-Mri Prediction](https://mlsanity.com/paper/2511.02558/) (2025)
- [Cortex-Grounded Diffusion Models for Brain Image Generation](https://mlsanity.com/paper/2601.19498/) (2026)
- [Generating Novel Brain Morphology by Deforming Learned Templates](https://mlsanity.com/paper/2503.03778/) (2025)

## Abstract (arXiv)

We describe Countersynth, a conditional generative model of diffeomorphic deformations that induce label-driven, biologically plausible changes in volumetric brain images. The model is intended to synthesise counterfactual training data augmentations for downstream discriminative modelling tasks where fidelity is limited by data imbalance, distributional instability, confounding, or underspecification, and exhibits inequitable performance across distinct subpopulations. Focusing on demographic attributes, we evaluate the quality of synthesized counterfactuals with voxel-based morphometry, classification and regression of the conditioning attributes, and the Fr'echet inception distance. Examining downstream discriminative performance in the context of engineered demographic imbalance and confounding, we use UK Biobank magnetic resonance imaging data to benchmark CounterSynth augmentation against current solutions to these problems. We achieve state-of-the-art improvements, both in overall fidelity and equity. The source code for CounterSynth is available online.