# DECAR: Deep Clustering for learning general-purpose Audio Representations

arXiv: https://arxiv.org/abs/2110.08895 · HTML: https://mlsanity.com/paper/2110.08895/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 55 references that are in the map.
- First posted on arXiv 17 Oct 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)

## Built on by (most cited first, 7 of 7)

- [Benchmarking Representations for Speech, Music, and Acoustic Events](https://mlsanity.com/paper/2405.00934/) (2024)
- [Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data](https://mlsanity.com/paper/2410.02056/) (2024)
- [SLICER: Learning universal audio representations using low-resource self-supervised pre-training](https://mlsanity.com/paper/2211.01519/) (2022)
- [MAST: Multiscale Audio Spectrogram Transformers](https://mlsanity.com/paper/2211.01515/) (2022)
- [FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous Self-Supervised Learning](https://mlsanity.com/paper/2312.13026/) (2023)
- [Evaluation of Deep Audio Representations for Hearables](https://mlsanity.com/paper/2502.06664/) (2025)
- [UNFUSED: UNsupervised Finetuning Using SElf supervised Distillation](https://mlsanity.com/paper/2303.05668/) (2023)

## Abstract (arXiv)

We introduce DECAR, a self-supervised pre-training approach for learning general-purpose audio representations. Our system is based on clustering: it utilizes an offline clustering step to provide target labels that act as pseudo-labels for solving a prediction task. We develop on top of recent advances in self-supervised learning for computer vision and design a lightweight, easy-to-use self-supervised pre-training scheme. We pre-train DECAR embeddings on a balanced subset of the large-scale Audioset dataset and transfer those representations to 9 downstream classification tasks, including speech, music, animal sounds, and acoustic scenes. Furthermore, we conduct ablation studies identifying key design choices and also make all our code and pre-trained models publicly available.