# Counterfactual Generative Networks

arXiv: https://arxiv.org/abs/2101.06046 · HTML: https://mlsanity.com/paper/2101.06046/

## Key facts

- Cited by 111 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 53 references that are in the map.
- First posted on arXiv 15 Jan 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training](https://mlsanity.com/paper/1709.02023/) (2017)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness](https://mlsanity.com/paper/1811.00007/) (2018)

## Built on by (most cited first, 30 of 111)

- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)
- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [Diffusion Causal Models for Counterfactual Estimation](https://mlsanity.com/paper/2202.10166/) (2022)
- [Diffusion Models for Counterfactual Explanations](https://mlsanity.com/paper/2203.15636/) (2022)
- [Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?](https://mlsanity.com/paper/2106.02890/) (2021)
- [FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimization](https://mlsanity.com/paper/2112.01573/) (2021)
- [MaskTune: Mitigating Spurious Correlations by Forcing to Explore](https://mlsanity.com/paper/2210.00055/) (2022)
- [Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations](https://mlsanity.com/paper/2103.10226/) (2021)
- [A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes](https://mlsanity.com/paper/2201.10766/) (2022)
- [Causal Component Analysis](https://mlsanity.com/paper/2305.17225/) (2023)
- [Ensembling with Deep Generative Views](https://mlsanity.com/paper/2104.14551/) (2021)
- [LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images](https://mlsanity.com/paper/2305.19164/) (2023)
- [A Causal Lens for Controllable Text Generation](https://mlsanity.com/paper/2201.09119/) (2022)
- [Finding and Fixing Spurious Patterns with Explanations](https://mlsanity.com/paper/2106.02112/) (2021)
- [CausalAdv: Adversarial Robustness through the Lens of Causality](https://mlsanity.com/paper/2106.06196/) (2021)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Measuring axiomatic soundness of counterfactual image models](https://mlsanity.com/paper/2303.01274/) (2023)
- [Cycle-Consistent Counterfactuals by Latent Transformations](https://mlsanity.com/paper/2203.15064/) (2022)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [D'ARTAGNAN: Counterfactual Video Generation](https://mlsanity.com/paper/2206.01651/) (2022)
- [Towards Explaining Distribution Shifts](https://mlsanity.com/paper/2210.10275/) (2022)
- [ComCLIP: Training-Free Compositional Image and Text Matching](https://mlsanity.com/paper/2211.13854/) (2022)
- [Certified Robustness Against Natural Language Attacks by Causal Intervention](https://mlsanity.com/paper/2205.12331/) (2022)
- [Zero-shot Model Diagnosis](https://mlsanity.com/paper/2303.15441/) (2023)
- [From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling](https://mlsanity.com/paper/2310.11011/) (2023)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [Counterfactual Image Editing](https://mlsanity.com/paper/2403.09683/) (2024)
- [Towards Counterfactual Image Manipulation via CLIP](https://mlsanity.com/paper/2207.02812/) (2022)
- [Partial Identification of Treatment Effects with Implicit Generative Models](https://mlsanity.com/paper/2210.08139/) (2022)

## Abstract (arXiv)

Neural networks are prone to learning shortcuts -- they often model simple correlations, ignoring more complex ones that potentially generalize better. Prior works on image classification show that instead of learning a connection to object shape, deep classifiers tend to exploit spurious correlations with low-level texture or the background for solving the classification task. In this work, we take a step towards more robust and interpretable classifiers that explicitly expose the task's causal structure. Building on current advances in deep generative modeling, we propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background; hence, they allow for generating counterfactual images. We demonstrate the ability of our model to generate such images on MNIST and ImageNet. Further, we show that the counterfactual images can improve out-of-distribution robustness with a marginal drop in performance on the original classification task, despite being synthetic. Lastly, our generative model can be trained efficiently on a single GPU, exploiting common pre-trained models as inductive biases.