mlsanityData to 7 Oct 2026

Counterfactual Image Editing

arXiv 2403.09683 ↗First posted 7 Feb 2024cs.CV cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. The current literature on the topic focuses primarily on changing individual features while remaining silent about the causal relationships between these features, as present in the real world. In this paper, we formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images through a special type of model called augmented structural causal models (ASCMs). Second, we show two fundamental impossibility results: (1) counterfactual editing is impossible from i.i.d. image samples and their corresponding labels alone; (2) even when the causal relationships between the latent generative factors and images are available, no guarantees regarding the output of the model can be provided. Third, we propose a relaxation for this challenging problem by approximating non-identifiable counterfactual distributions with a new family of counterfactual-consistent estimators. This family exhibits the desirable property of preserving features that the user cares about across both factual and counterfactual worlds. Finally, we develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 65 of its 109 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

65 of 109 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2016Layer Normalizationcited by 5,579
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
38 more
  1. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  2. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  3. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  4. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  5. 2018Self-Attention Generative Adversarial Networkscited by 809
  6. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  7. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  8. 2018Disentangling by Factorisingcited by 614
  9. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  10. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  11. 2016Adversarial Feature Learningcited by 350
  12. 2019Counterfactual Visual Explanationscited by 312
  13. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  14. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  15. 2013Direct and Indirect Effectscited by 276
  16. 2018Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negativescited by 261
  17. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  18. 2019Interpretable Counterfactual Explanations Guided by Prototypescited by 223
  19. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 221
  20. 2016Adversarially Learned Inferencecited by 216
  21. 2020Deep Structural Causal Models for Tractable Counterfactual Inferencecited by 210
  22. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  23. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  24. 2020Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Reviewcited by 174
  25. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 158
  26. 2017CausalGAN: Learning Causal Implicit Generative Models with Adversarial Trainingcited by 152
  27. 2022Weakly supervised causal representation learningcited by 140
  28. 2019Explaining Classifiers with Causal Concept Effect (CaCE)cited by 135
  29. 2019Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systemscited by 128
  30. 2021Counterfactual Generative Networkscited by 111
  31. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  32. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latentscited by 103
  33. 2021The Causal-Neural Connection: Expressiveness, Learnability, and Inferencecited by 97
  34. 2022Diffusion Visual Counterfactual Explanationscited by 96
  35. 2022Diffusion Causal Models for Counterfactual Estimationcited by 87
  36. 2023Identifiability Guarantees for Causal Disentanglement from Soft Interventionscited by 84
  37. 2020Causal Interpretability for Machine Learning -- Problems, Methods and Evaluationcited by 82
  38. 2019Permutation-Based Causal Structure Learning with Unknown Intervention Targetscited by 69

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025Diffusion Counterfactual Generation with Semantic Abductioncited by 8
  2. 2024Causality from Bottom to Top: A Surveycited by 5
  3. 2025Counterfactual Identifiability via Dynamic Optimal Transportcited by 5
  4. 2025Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generationcited by 4
  5. 2025CausalAffect: Causal Discovery for Facial Affective Understandingcited by 3
  6. 2025Locally Explaining Prediction Behavior via Gradual Interventions and Measuring Property Gradientscited by 2
  7. 2025Segmentor-Guided Counterfactual Fine-Tuning for Locally Coherent and Targeted Image Synthesiscited by 2
  8. 2025Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imagingcited by 1
  9. 2025Flow IV: Counterfactual Inference In Nonseparable Outcome Models Using Instrumental Variablescited by 1
  10. 2025RoentMod: A Synthetic Chest X-Ray Modification Model to Identify and Correct Image Interpretation Model Shortcutscited by 1
  11. 2025Towards Causal Representation Learning with Observable Sources as Auxiliariescited by 1
  12. 2026Generating Chest X-Ray Counterfactuals by Specialising Foundation Image Modelscited by 1
8 more
  1. 2025Leveraging a Simulator for Learning Causal Representations from Post-Treatment Covariates for CATEcited by 0
  2. 2025Unsupervised Structural-Counterfactual Generation under Domain Shiftcited by 0
  3. 2025Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradientscited by 0
  4. 2025From Black-box to Causal-box: Towards Building More Interpretable Modelscited by 0
  5. 2025Causal Regime Detection in Energy Markets With Augmented Time Series Structural Causal Modelscited by 0
  6. 2026Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesiscited by 0
  7. 2026Causal Variational Deep Embedding: A Family of Interventional Generators for Confounded Imagescited by 0
  8. 2026Common-Witness Certificates and Sharp Feature Bounds for Counterfactual Image Auditingcited by 0