mlsanityData to 14 Oct 2026

Counterfactual Generative Networks

arXiv 2101.06046 ↗First posted 15 Jan 2021cs.LG cs.CVcited by 111

Not on the map. Why?

Abstract (arXiv)

Neural networks are prone to learning shortcuts -- they often model simple correlations, ignoring more complex ones that potentially generalize better. Prior works on image classification show that instead of learning a connection to object shape, deep classifiers tend to exploit spurious correlations with low-level texture or the background for solving the classification task. In this work, we take a step towards more robust and interpretable classifiers that explicitly expose the task's causal structure. Building on current advances in deep generative modeling, we propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background; hence, they allow for generating counterfactual images. We demonstrate the ability of our model to generate such images on MNIST and ImageNet. Further, we show that the counterfactual images can improve out-of-distribution robustness with a marginal drop in performance on the original classification task, despite being synthetic. Lastly, our generative model can be trained efficiently on a single GPU, exploiting common pre-trained models as inductive biases.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 15 Jan 2021.

Builds on

39 of 53 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  10. 2019Invariant Risk Minimizationcited by 1,854
  11. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
27 more
  1. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  3. 2018Recognition in Terra Incognitacited by 658
  4. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  5. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  6. 2015A Neural Algorithm of Artistic Stylecited by 483
  7. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  8. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 412
  9. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  10. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 289
  11. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  12. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  13. 2019Causality for Machine Learningcited by 243
  14. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  15. 2017CausalGAN: Learning Causal Implicit Generative Models with Adversarial Trainingcited by 152
  16. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  17. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  18. 2018Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustnesscited by 108
  19. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  20. 2020Controlling generative models with continuous factors of variationscited by 93
  21. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 92
  22. 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
  23. 2018Counterfactuals uncover the modular structure of deep generative modelscited by 51
  24. 2020Towards causal generative scene models via competition of expertscited by 37
  25. 2017LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generationcited by 36
  26. 2019MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generationcited by 29
  27. 2020Selecting Data Augmentation for Simulating Interventionscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 31; 2023: 21; 2024: 16; 2025: 18; 2026: 11.

111 papers.

Sort:

By area:

  1. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  2. 2021Projected GANs Converge Fastercited by 208
  3. 2022Diffusion Causal Models for Counterfactual Estimationcited by 87
  4. 2022Diffusion Models for Counterfactual Explanationscited by 79
  5. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  6. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  7. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
  8. 2021Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanationscited by 56
  9. 2022A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributescited by 56
  10. 2023Causal Component Analysiscited by 56
  11. 2021Ensembling with Deep Generative Viewscited by 53
  12. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
38 more
  1. 2022A Causal Lens for Controllable Text Generationcited by 45
  2. 2021Finding and Fixing Spurious Patterns with Explanationscited by 38
  3. 2021CausalAdv: Adversarial Robustness through the Lens of Causalitycited by 37
  4. 2024Foundation Models for Music: A Surveycited by 37
  5. 2023Measuring axiomatic soundness of counterfactual image modelscited by 34
  6. 2022Cycle-Consistent Counterfactuals by Latent Transformationscited by 33
  7. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  8. 2022D'ARTAGNAN: Counterfactual Video Generationcited by 28
  9. 2022Towards Explaining Distribution Shiftscited by 26
  10. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
  11. 2022Certified Robustness Against Natural Language Attacks by Causal Interventioncited by 24
  12. 2023Zero-shot Model Diagnosiscited by 23
  13. 2023From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modelingcited by 22
  14. 2024Benchmarking Counterfactual Image Generationcited by 22
  15. 2023Causal Deep Learningcited by 20
  16. 2024Counterfactual Image Editingcited by 20
  17. 2022Towards Counterfactual Image Manipulation via CLIPcited by 19
  18. 2022Partial Identification of Treatment Effects with Implicit Generative Modelscited by 17
  19. 2023Counterfactual Generative Models for Time-Varying Treatmentscited by 17
  20. 2023Latent Diffusion Counterfactual Explanationscited by 17
  21. 2022On the link between conscious function and general intelligence in humans and machinescited by 16
  22. 2021Designing Counterfactual Generators using Deep Model Inversioncited by 15
  23. 2021Learn-Explain-Reinforce: Counterfactual Reasoning and Its Guidance to Reinforce an Alzheimer's Disease Diagnosis Modelcited by 14
  24. 2023Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Modelcited by 14
  25. 2021Empowering Language Understanding with Counterfactual Reasoningcited by 13
  26. 2022Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Datasetcited by 13
  27. 2024Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesiscited by 13
  28. 2021Causal Discovery from Conditionally Stationary Time Seriescited by 12
  29. 2022Deep Structural Causal Shape Modelscited by 11
  30. 2024Unsupervised Concept Discovery Mitigates Spurious Correlationscited by 11
  31. 2024Counterfactual Generative Modeling with Variational Causal Inferencecited by 11
  32. 2025Are We Done with Object-Centric Learning?cited by 11
  33. 2021DiagViB-6: A Diagnostic Benchmark Suite for Vision Models in the Presence of Shortcut and Generalization Opportunitiescited by 10
  34. 2022Making a (Counterfactual) Difference One Rationale at a Timecited by 10
  35. 2022A Survey of Deep Causal Models and Their Industrial Applicationscited by 10
  36. 2022GOOD: Exploring Geometric Cues for Detecting Objects in an Open Worldcited by 10
  37. 2023Emerging Synergies in Causality and Deep Generative Models: A Surveycited by 10
  38. 2023Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularizationcited by 10

50 most cited of 111 All 111 (JSON)