# Visually Guided Self Supervised Learning of Speech Representations

arXiv: https://arxiv.org/abs/2001.04316 · HTML: https://mlsanity.com/paper/2001.04316/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2020.

## Built on by (most cited first, 8 of 8)

- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Machine Learning for Stuttering Identification: Review, Challenges and Future Directions](https://mlsanity.com/paper/2107.04057/) (2021)
- [SepTr: Separable Transformer for Audio Spectrogram Processing](https://mlsanity.com/paper/2203.09581/) (2022)
- [Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition](https://mlsanity.com/paper/2203.07996/) (2022)
- [Self-paced ensemble learning for speech and audio classification](https://mlsanity.com/paper/2103.11988/) (2021)
- [Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Prediction](https://mlsanity.com/paper/2311.02898/) (2023)
- [Universal Facial Encoding of Codec Avatars from VR Headsets](https://mlsanity.com/paper/2407.13038/) (2024)
- [Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanism](https://mlsanity.com/paper/2106.11769/) (2021)

## Abstract (arXiv)

Self supervised representation learning has recently attracted a lot of research interest for both the audio and visual modalities. However, most works typically focus on a particular modality or feature alone and there has been very limited work that studies the interaction between the two modalities for learning self supervised representations. We propose a framework for learning audio representations guided by the visual modality in the context of audiovisual speech. We employ a generative audio-to-video training scheme in which we animate a still image corresponding to a given audio clip and optimize the generated video to be as close as possible to the real video of the speech segment. Through this process, the audio encoder network learns useful speech representations that we evaluate on emotion recognition and speech recognition. We achieve state of the art results for emotion recognition and competitive results for speech recognition. This demonstrates the potential of visual supervision for learning audio representations as a novel way for self-supervised learning which has not been explored in the past. The proposed unsupervised audio features can leverage a virtually unlimited amount of training data of unlabelled audiovisual speech and have a large number of potentially promising applications.