# Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables

arXiv: https://arxiv.org/abs/2609.10778 · HTML: https://mlsanity.com/paper/2609.10778/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation](https://mlsanity.com/paper/1711.01468/) (2017)
- [High Fidelity Image Counterfactuals with Probabilistic Causal Models](https://mlsanity.com/paper/2306.15764/) (2023)
- [Analyzing Overfitting under Class Imbalance in Neural Networks for Image Segmentation](https://mlsanity.com/paper/2102.10365/) (2021)
- [Addressing Fairness Issues in Deep Learning-Based Medical Image Analysis: A Systematic Review](https://mlsanity.com/paper/2209.13177/) (2022)
- [Understanding Test-Time Augmentation](https://mlsanity.com/paper/2402.06892/) (2024)
- [No Fair Lunch: A Causal Perspective on Dataset Bias in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2307.16526/) (2023)
- [Semi-Supervised Learning for Deep Causal Generative Models](https://mlsanity.com/paper/2403.18717/) (2024)

## Abstract (arXiv)

Machine learning models can achieve strong test performance while relying on demographic or acquisition-related shortcuts. We propose counterfactual (CF) marginalisation as a test-time evaluation procedure for assessing robustness of classification models to such variables. Given a CF image generator, we intervene on nuisance parent variables such as age or sex, generate CF versions of each test image, and average predictions over a target intervention distribution. This produces intervention-aware predictions that marginalise demographic effects while preserving patient-specific latent information. We use these predictions to define metrics for CF risk, calibration, stability and worst-case sensitivity. We demonstrate this framework's utility for quantitative robustness evaluation.