# Counterfactual contrastive learning: robust representations via causal image synthesis

arXiv: https://arxiv.org/abs/2403.09605 · HTML: https://mlsanity.com/paper/2403.09605/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Diffusion Visual Counterfactual Explanations](https://mlsanity.com/paper/2210.11841/) (2022)
- [Evaluating and Mitigating Bias in Image Classifiers: A Causal Perspective Using Counterfactuals](https://mlsanity.com/paper/2009.08270/) (2020)
- [High Fidelity Image Counterfactuals with Probabilistic Causal Models](https://mlsanity.com/paper/2306.15764/) (2023)
- [What is Healthy? Generative Counterfactual Diffusion for Lesion Localization](https://mlsanity.com/paper/2207.12268/) (2022)
- [Measuring axiomatic soundness of counterfactual image models](https://mlsanity.com/paper/2303.01274/) (2023)
- [Self-supervised Learning from 100 Million Medical Images](https://mlsanity.com/paper/2201.01283/) (2022)
- [Diffusion Models for Counterfactual Generation and Anomaly Detection in Brain Images](https://mlsanity.com/paper/2308.02062/) (2023)
- [CheXplaining in Style: Counterfactual Explanations for Chest X-rays using StyleGAN](https://mlsanity.com/paper/2207.07553/) (2022)
- [Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning](https://mlsanity.com/paper/2207.00148/) (2022)
- [High-resolution synthesis of high-density breast mammograms: Application to improved fairness in deep learning based mass detection](https://mlsanity.com/paper/2209.09809/) (2022)
- [Inherently Interpretable Multi-Label Classification Using Class-Specific Counterfactuals](https://mlsanity.com/paper/2303.00500/) (2023)
- [Adversarial Counterfactual Augmentation: Application in Alzheimer's Disease Classification](https://mlsanity.com/paper/2203.07815/) (2022)
- [Counterfactual Explanation of Brain Activity Classifiers using Image-to-Image Transfer by Generative Adversarial Network](https://mlsanity.com/paper/2110.14927/) (2021)

## Built on by (most cited first, 11 of 11)

- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Diffusion Counterfactual Generation with Semantic Abduction](https://mlsanity.com/paper/2506.07883/) (2025)
- [CF-Seg: Counterfactuals meet Segmentation](https://mlsanity.com/paper/2506.16213/) (2025)
- [Mask of truth: model sensitivity to unexpected regions of medical images](https://mlsanity.com/paper/2412.04030/) (2024)
- [Counterfactual Identifiability via Dynamic Optimal Transport](https://mlsanity.com/paper/2510.08294/) (2025)
- [Segmentor-Guided Counterfactual Fine-Tuning for Locally Coherent and Targeted Image Synthesis](https://mlsanity.com/paper/2509.24913/) (2025)
- [Causal Representation Learning with Observational Grouping for CXR Classification](https://mlsanity.com/paper/2506.20582/) (2025)
- [Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Models](https://mlsanity.com/paper/2609.24879/) (2026)
- [Learning to Harmonize Cross-vendor X-ray Images by Non-linear Image Dynamics Correction](https://mlsanity.com/paper/2504.10080/) (2025)
- [Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis](https://mlsanity.com/paper/2603.21213/) (2026)
- [TypedCSIP: Typed Counterfactual Pretraining for Chinese Legislative Conflict Classification](https://mlsanity.com/paper/2605.25474/) (2026)

## Abstract (arXiv)

Contrastive pretraining is well-known to improve downstream task performance and model generalisation, especially in limited label settings. However, it is sensitive to the choice of augmentation pipeline. Positive pairs should preserve semantic information while destroying domain-specific information. Standard augmentation pipelines emulate domain-specific changes with pre-defined photometric transformations, but what if we could simulate realistic domain changes instead? In this work, we show how to utilise recent progress in counterfactual image generation to this effect. We propose CF-SimCLR, a counterfactual contrastive learning approach which leverages approximate counterfactual inference for positive pair creation. Comprehensive evaluation across five datasets, on chest radiography and mammography, demonstrates that CF-SimCLR substantially improves robustness to acquisition shift with higher downstream performance on both in- and out-of-distribution data, particularly for domains which are under-represented during training.