# EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data

arXiv: https://arxiv.org/abs/2409.07566 · HTML: https://mlsanity.com/paper/2409.07566/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography](https://mlsanity.com/paper/1908.06948/) (2019)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model](https://mlsanity.com/paper/1706.03458/) (2017)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Knowledge Distillation for Multi-task Learning](https://mlsanity.com/paper/2007.06889/) (2020)
- [Ultrasound Video Transformers for Cardiac Ejection Fraction Estimation](https://mlsanity.com/paper/2107.00977/) (2021)
- [Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis](https://mlsanity.com/paper/2303.12644/) (2023)
- [Knowledge Distillation for Action Anticipation via Label Smoothing](https://mlsanity.com/paper/2004.07711/) (2020)
- [EchoNet-Synthetic: Privacy-preserving Video Generation for Safe Medical Data Sharing](https://mlsanity.com/paper/2406.00808/) (2024)
- [EchoGNN: Explainable Ejection Fraction Estimation with Graph Neural Networks](https://mlsanity.com/paper/2208.14003/) (2022)
- [EchoCoTr: Estimation of the Left Ventricular Ejection Fraction from Spatiotemporal Echocardiography](https://mlsanity.com/paper/2209.04242/) (2022)
- [Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning](https://mlsanity.com/paper/2308.09544/) (2023)
- [Vision Transformers Are Good Mask Auto-Labelers](https://mlsanity.com/paper/2301.03992/) (2023)
- [Recurrent Aggregation Learning for Multi-View Echocardiographic Sequences Segmentation](https://mlsanity.com/paper/1907.11292/) (2019)
- [Hierarchical Vision Transformers for Cardiac Ejection Fraction Estimation](https://mlsanity.com/paper/2304.00177/) (2023)
- [Lightweight and Interpretable Left Ventricular Ejection Fraction Estimation using Mobile U-Net](https://mlsanity.com/paper/2304.07951/) (2023)

## Built on by (most cited first, 1 of 1)

- [A DyL-Unet framework based on dynamic learning for Temporally Consistent Echocardiographic Segmentation](https://mlsanity.com/paper/2509.19052/) (2025)

## Abstract (arXiv)

The application of machine learning to medical ultrasound videos of the heart, i.e., echocardiography, has recently gained traction with the availability of large public datasets. Traditional supervised tasks, such as ejection fraction regression, are now making way for approaches focusing more on the latent structure of data distributions, as well as generative methods. We propose a model trained exclusively by knowledge distillation, either on real or synthetical data, involving retrieving masks suggested by a teacher model. We achieve state-of-the-art (SOTA) values on the task of identifying end-diastolic and end-systolic frames. By training the model only on synthetic data, it reaches segmentation capabilities close to the performance when trained on real data with a significantly reduced number of weights. A comparison with the 5 main existing methods shows that our method outperforms the others in most cases. We also present a new evaluation method that does not require human annotation and instead relies on a large auxiliary model. We show that this method produces scores consistent with those obtained from human annotations. Relying on the integrated knowledge from a vast amount of records, this method overcomes certain inherent limitations of human annotator labeling. Code: https://github.com/GregoirePetit/EchoDFKD