# D'ARTAGNAN: Counterfactual Video Generation

arXiv: https://arxiv.org/abs/2206.01651 · HTML: https://mlsanity.com/paper/2206.01651/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 3 Jun 2022.

## Builds on (most cited first, 16 of 16)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training](https://mlsanity.com/paper/1709.02023/) (2017)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Counterfactual Generative Networks](https://mlsanity.com/paper/2101.06046/) (2021)
- [Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning](https://mlsanity.com/paper/1809.10780/) (2018)
- [Perfect Match: A Simple Method for Learning Representations For Counterfactual Inference With Neural Networks](https://mlsanity.com/paper/1810.00656/) (2018)
- [Counterfactual Representation Learning with Balancing Weights](https://mlsanity.com/paper/2010.12618/) (2020)
- [Ultrasound Video Transformers for Cardiac Ejection Fraction Estimation](https://mlsanity.com/paper/2107.00977/) (2021)
- [Content-Preserving Unpaired Translation from Simulated to Realistic Ultrasound Images](https://mlsanity.com/paper/2103.05745/) (2021)
- [XCAT-GAN for Synthesizing 3D Consistent Labeled Cardiac MR Images on Anatomically Variable XCAT Phantoms](https://mlsanity.com/paper/2007.13408/) (2020)
- [4D Semantic Cardiac Magnetic Resonance Image Synthesis on XCAT Anatomical Model](https://mlsanity.com/paper/2002.07089/) (2020)
- [A nonparametric projection-based estimator for the probability of causation, with application to water sanitation in Kenya](https://mlsanity.com/paper/1810.00767/) (2018)

## Built on by (most cited first, 28 of 28)

- [Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis](https://mlsanity.com/paper/2303.12644/) (2023)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [A Review of Causality for Learning Algorithms in Medical Image Analysis](https://mlsanity.com/paper/2206.05498/) (2022)
- [EchoNet-Synthetic: Privacy-preserving Video Generation for Safe Medical Data Sharing](https://mlsanity.com/paper/2406.00808/) (2024)
- [A personalized time-resolved 3D mesh generative model for unveiling normal heart dynamics](https://mlsanity.com/paper/2409.13825/) (2024)
- [Counterfactual Generative Models for Time-Varying Treatments](https://mlsanity.com/paper/2305.15742/) (2023)
- [Causal models in string diagrams](https://mlsanity.com/paper/2304.07638/) (2023)
- [DreaMR: Diffusion-driven Counterfactual Explanation for Functional MRI](https://mlsanity.com/paper/2307.09547/) (2023)
- [CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy](https://mlsanity.com/paper/2301.13098/) (2023)
- [Ultrasound Image-to-Video Synthesis via Latent Dynamic Diffusion Models](https://mlsanity.com/paper/2503.14966/) (2025)
- [Spatio-temporal neural distance fields for conditional generative modeling of the heart](https://mlsanity.com/paper/2407.10663/) (2024)
- [Contrastive representations of high-dimensional, structured treatments](https://mlsanity.com/paper/2411.19245/) (2024)
- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [Non-parametric identifiability and sensitivity analysis of synthetic control models](https://mlsanity.com/paper/2301.07656/) (2023)
- [DiffExplainer: Unveiling Black Box Models Via Counterfactual Generation](https://mlsanity.com/paper/2406.15182/) (2024)
- [EchoFlow: A Foundation Model for Cardiac Ultrasound Image and Video Generation](https://mlsanity.com/paper/2503.22357/) (2025)
- [Spillover Detection for Donor Selection in Synthetic Control Models](https://mlsanity.com/paper/2406.11399/) (2024)
- [Explainable and Controllable Motion Curve Guided Cardiac Ultrasound Video Generation](https://mlsanity.com/paper/2407.21490/) (2024)
- [Uncovering Hidden Subspaces in Video Diffusion Models Using Re-Identification](https://mlsanity.com/paper/2411.04956/) (2024)
- [Nature versus nurture in galaxy formation: the effect of environment on star formation with causal machine learning](https://mlsanity.com/paper/2412.02439/) (2024)
- [LD-ViCE: Latent Diffusion Model for Video Counterfactual Explanations](https://mlsanity.com/paper/2509.08422/) (2025)
- [Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanations](https://mlsanity.com/paper/2511.20295/) (2025)
- [Exploring the Hyperparameter Space of Image Diffusion Models for Echocardiogram Generation](https://mlsanity.com/paper/2311.01567/) (2023)
- [Label-free Motion-Conditioned Diffusion Model for Cardiac Ultrasound Synthesis](https://mlsanity.com/paper/2512.09418/) (2025)
- [InfoMotion: A Graph-Based Approach to Video Dataset Distillation for Echocardiography](https://mlsanity.com/paper/2512.09422/) (2025)
- [Quantum oracles give an advantage for identifying classical counterfactuals](https://mlsanity.com/paper/2512.13692/) (2025)
- [Causal Transfer in Medical Image Analysis](https://mlsanity.com/paper/2603.24388/) (2026)
- [Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2610.13504/) (2026)

## Abstract (arXiv)

Causally-enabled machine learning frameworks could help clinicians to identify the best course of treatments by answering counterfactual questions. We explore this path for the case of echocardiograms by looking into the variation of the Left Ventricle Ejection Fraction, the most essential clinical metric gained from these examinations. We combine deep neural networks, twin causal networks and generative adversarial methods for the first time to build D'ARTAGNAN (Deep ARtificial Twin-Architecture GeNerAtive Networks), a novel causal generative model. We demonstrate the soundness of our approach on a synthetic dataset before applying it to cardiac ultrasound videos to answer the question: "What would this echocardiogram look like if the patient had a different ejection fraction?". To do so, we generate new ultrasound videos, retaining the video style and anatomy of the original patient, while modifying the Ejection Fraction conditioned on a given input. We achieve an SSIM score of 0.79 and an R2 score of 0.51 on the counterfactual videos. Code and models are available at: https://github.com/HReynaud/dartagnan.