# Self-supervised Contrastive Video-Speech Representation Learning for Ultrasound

arXiv: https://arxiv.org/abs/2008.06607 · HTML: https://mlsanity.com/paper/2008.06607/

## Key facts

- Cited by 15 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 14 Aug 2020.

## Built on by (most cited first, 15 of 15)

- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Positional Contrastive Learning for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2106.09157/) (2021)
- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [Lesion-based Contrastive Learning for Diabetic Retinopathy Grading from Fundus Images](https://mlsanity.com/paper/2107.08274/) (2021)
- [Contrastive Attraction and Contrastive Repulsion for Representation Learning](https://mlsanity.com/paper/2105.03746/) (2021)
- [A Review of Predictive and Contrastive Self-supervised Learning for Medical Images](https://mlsanity.com/paper/2302.05043/) (2023)
- [Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasound](https://mlsanity.com/paper/2208.10642/) (2022)
- [Contrastive Learning for View Classification of Echocardiograms](https://mlsanity.com/paper/2108.03124/) (2021)
- [HiCo: Hierarchical Contrastive Learning for Ultrasound Video Model Pretraining](https://mlsanity.com/paper/2210.04477/) (2022)
- [Generating and Weighting Semantically Consistent Sample Pairs for Ultrasound Contrastive Learning](https://mlsanity.com/paper/2212.04097/) (2022)
- [Visually-aware Acoustic Event Detection using Heterogeneous Graphs](https://mlsanity.com/paper/2207.07935/) (2022)
- [Voice-assisted Image Labelling for Endoscopic Ultrasound Classification using Neural Networks](https://mlsanity.com/paper/2110.06367/) (2021)
- [Source Identification: A Self-Supervision Task for Dense Prediction](https://mlsanity.com/paper/2307.02238/) (2023)
- [Show from Tell: Audio-Visual Modelling in Clinical Settings](https://mlsanity.com/paper/2310.16477/) (2023)
- [CM-UNet: A Self-Supervised Learning-Based Model for Coronary Artery Segmentation in X-Ray Angiography](https://mlsanity.com/paper/2507.17779/) (2025)

## Abstract (arXiv)

In medical imaging, manual annotations can be expensive to acquire and sometimes infeasible to access, making conventional deep learning-based models difficult to scale. As a result, it would be beneficial if useful representations could be derived from raw data without the need for manual annotations. In this paper, we propose to address the problem of self-supervised representation learning with multi-modal ultrasound video-speech raw data. For this case, we assume that there is a high correlation between the ultrasound video and the corresponding narrative speech audio of the sonographer. In order to learn meaningful representations, the model needs to identify such correlation and at the same time understand the underlying anatomical features. We designed a framework to model the correspondence between video and audio without any kind of human annotations. Within this framework, we introduce cross-modal contrastive learning and an affinity-aware self-paced learning scheme to enhance correlation modelling. Experimental evaluations on multi-modal fetal ultrasound video and audio show that the proposed approach is able to learn strong representations and transfers well to downstream tasks of standard plane detection and eye-gaze prediction.