# Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning

arXiv: https://arxiv.org/abs/2010.13991 · HTML: https://mlsanity.com/paper/2010.13991/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 30 of 34)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [Sound Localization by Self-Supervised Time Delay Estimation](https://mlsanity.com/paper/2204.12489/) (2022)
- [CR-CTC: Consistency regularization on CTC for improved speech recognition](https://mlsanity.com/paper/2410.05101/) (2024)
- [Exploring Self-Supervised Representation Ensembles for COVID-19 Cough Classification](https://mlsanity.com/paper/2105.07566/) (2021)
- [That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation](https://mlsanity.com/paper/2210.01116/) (2022)
- [SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks](https://mlsanity.com/paper/2408.13040/) (2024)
- [CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representations](https://mlsanity.com/paper/2210.02592/) (2022)
- [Equivariant Self-Supervision for Musical Tempo Estimation](https://mlsanity.com/paper/2209.01478/) (2022)
- [Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data](https://mlsanity.com/paper/2202.00097/) (2022)
- [SampleMatch: Drum Sample Retrieval by Musical Context](https://mlsanity.com/paper/2208.01141/) (2022)
- [Pretext Tasks selection for multitask self-supervised speech representation learning](https://mlsanity.com/paper/2107.00594/) (2021)
- [Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis](https://mlsanity.com/paper/2304.00395/) (2023)
- [data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setup](https://mlsanity.com/paper/2211.01246/) (2022)
- [Computer Vision Self-supervised Learning Methods on Time Series](https://mlsanity.com/paper/2109.00783/) (2021)
- [Conditional independence for pretext task selection in Self-supervised speech representation learning](https://mlsanity.com/paper/2104.07388/) (2021)
- [ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks](https://mlsanity.com/paper/2407.21066/) (2024)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognition](https://mlsanity.com/paper/2203.16973/) (2022)
- [Contrastive Siamese Network for Semi-supervised Speech Recognition](https://mlsanity.com/paper/2205.14054/) (2022)
- [Discovering COVID-19 Coughing and Breathing Patterns from Unlabeled Data Using Contrastive Learning with Varying Pre-Training Domains](https://mlsanity.com/paper/2306.01864/) (2023)
- [Optimizing Audio Augmentations for Contrastive Learning of Health-Related Acoustic Signals](https://mlsanity.com/paper/2309.05843/) (2023)
- [Multi-Variant Consistency based Self-supervised Learning for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2112.12522/) (2021)
- [TriNet: stabilizing self-supervised learning from complete or slow collapse on ASR](https://mlsanity.com/paper/2301.00656/) (2022)
- [MADI: Inter-domain Matching and Intra-domain Discrimination for Cross-domain Speech Recognition](https://mlsanity.com/paper/2302.11224/) (2023)
- [Removing Speaker Information from Speech Representation using Variable-Length Soft Pooling](https://mlsanity.com/paper/2404.00856/) (2024)
- [DELULU: Discriminative Embedding Learning Using Latent Units for Speaker-Aware Self-Trained Speech Foundational Model](https://mlsanity.com/paper/2510.17662/) (2025)

## Abstract (arXiv)

Self-supervised visual pretraining has shown significant progress recently. Among those methods, SimCLR greatly advanced the state of the art in self-supervised and semi-supervised learning on ImageNet. The input feature representations for speech and visual tasks are both continuous, so it is natural to consider applying similar objective on speech representation learning. In this paper, we propose Speech SimCLR, a new self-supervised objective for speech representation learning. During training, Speech SimCLR applies augmentation on raw speech and its spectrogram. Its objective is the combination of contrastive loss that maximizes agreement between differently augmented samples in the latent space and reconstruction loss of input representation. The proposed method achieved competitive results on speech emotion recognition and speech recognition.