mlsanityData to 7 Oct 2026

DECAR: Deep Clustering for learning general-purpose Audio Representations

arXiv 2110.08895 ↗First posted 17 Oct 2021cs.SD cs.CL eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce DECAR, a self-supervised pre-training approach for learning general-purpose audio representations. Our system is based on clustering: it utilizes an offline clustering step to provide target labels that act as pseudo-labels for solving a prediction task. We develop on top of recent advances in self-supervised learning for computer vision and design a lightweight, easy-to-use self-supervised pre-training scheme. We pre-train DECAR embeddings on a balanced subset of the large-scale Audioset dataset and transfer those representations to 9 downstream classification tasks, including speech, music, animal sounds, and acoustic scenes. Furthermore, we conduct ablation studies identifying key design choices and also make all our code and pre-trained models publicly available.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 17 Oct 2021.

Builds on

45 of 55 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
33 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  3. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  4. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  5. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  6. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  7. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2021AST: Audio Spectrogram Transformercited by 634
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  12. 2017Look, Listen and Learncited by 452
  13. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  14. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  15. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  16. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  17. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  18. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  19. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  20. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  21. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  22. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  23. 2018A multi-device dataset for urban acoustic scene classificationcited by 75
  24. 2017Unsupervised Learning of Semantic Audio Representationscited by 44
  25. 2018Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challengecited by 44
  26. 2021A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learningcited by 26
  27. 2019Self-supervised audio representation learning for mobile devicescited by 17
  28. 2017auDeep: Unsupervised Learning of Representations from Audio with Deep Recurrent Neural Networkscited by 16
  29. 2020Effects of Word-frequency based Pre- and Post- Processings for Audio Captioningcited by 12
  30. 2021The History of Speech Recognition to the Year 2030cited by 10
  31. 2020The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimationcited by 8
  32. 2021Speech Representation Learning Combining Conformer CPC with Deep Cluster for the ZeroSpeech Challenge 2021cited by 7
  33. 2017Variational Autoencoders for Learning Latent Representations of Speech Emotion: A Preliminary Studycited by 5

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 2; 2025: 1.

7 papers.

  1. 2024Benchmarking Representations for Speech, Music, and Acoustic Eventscited by 14
  2. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  3. 2022SLICER: Learning universal audio representations using low-resource self-supervised pre-trainingcited by 4
  4. 2022MAST: Multiscale Audio Spectrogram Transformerscited by 3
  5. 2023FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous Self-Supervised Learningcited by 1
  6. 2025Evaluation of Deep Audio Representations for Hearablescited by 1
  7. 2023UNFUSED: UNsupervised Finetuning Using SElf supervised Distillationcited by 0