# Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks

arXiv: https://arxiv.org/abs/2110.07313 · HTML: https://mlsanity.com/paper/2110.07313/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)
- [A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling](https://mlsanity.com/paper/1810.09050/) (2018)
- [Multimodal Self-Supervised Learning of General Audio Representations](https://mlsanity.com/paper/2104.12807/) (2021)
- [ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/2106.01621/) (2021)
- [Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision](https://mlsanity.com/paper/1911.05894/) (2019)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Audio tagging with noisy labels and minimal supervision](https://mlsanity.com/paper/1906.02975/) (2019)
- [A Sequential Self Teaching Approach for Improving Generalization in Sound Event Recognition](https://mlsanity.com/paper/2007.00144/) (2020)
- [Self-Supervised Learning from Automatically Separated Sound Scenes](https://mlsanity.com/paper/2105.02132/) (2021)
- [Do sound event representations generalize to other audio tasks? A case study in audio transfer learning](https://mlsanity.com/paper/2106.11335/) (2021)

## Built on by (most cited first, 22 of 22)

- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [A-JEPA: Joint-Embedding Predictive Architecture Can Listen](https://mlsanity.com/paper/2311.15830/) (2023)
- [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://mlsanity.com/paper/2401.03497/) (2024)
- [Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks](https://mlsanity.com/paper/2306.04186/) (2023)
- [HeAR -- Health Acoustic Representations](https://mlsanity.com/paper/2403.02522/) (2024)
- [When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP](https://mlsanity.com/paper/2303.16166/) (2023)
- [ATST: Audio Representation Learning with Teacher-Student Transformer](https://mlsanity.com/paper/2204.12076/) (2022)
- [CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification](https://mlsanity.com/paper/2203.06760/) (2022)
- [Equivariant Self-Supervision for Musical Tempo Estimation](https://mlsanity.com/paper/2209.01478/) (2022)
- [SampleMatch: Drum Sample Retrieval by Musical Context](https://mlsanity.com/paper/2208.01141/) (2022)
- [Linear-Complexity Self-Supervised Learning for Speech Processing](https://mlsanity.com/paper/2407.13377/) (2024)
- [The Computation of Generalized Embeddings for Underwater Acoustic Target Recognition using Contrastive Learning](https://mlsanity.com/paper/2505.12904/) (2025)
- [A Study on Robustness to Perturbations for Representations of Environmental Sound](https://mlsanity.com/paper/2203.10425/) (2022)
- [Structured-Noise Masked Modeling for Video, Audio and Beyond](https://mlsanity.com/paper/2503.16311/) (2025)
- [ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning](https://mlsanity.com/paper/2507.02666/) (2025)
- [Cross-Attention is Half Explanation in Speech-to-Text Models](https://mlsanity.com/paper/2509.18010/) (2025)
- [AudioMosaic: Contrastive Masked Audio Representation Learning](https://mlsanity.com/paper/2605.14231/) (2026)
- [Make Some Noise: Towards LLM audio reasoning and generation using sound tokens](https://mlsanity.com/paper/2503.22275/) (2025)
- [Representation-Regularized Convolutional Audio Transformer for Audio Understanding](https://mlsanity.com/paper/2601.21612/) (2026)

## Abstract (arXiv)

Representation learning from unlabeled data has been of major interest in artificial intelligence research. While self-supervised speech representation learning has been popular in the speech research community, very few works have comprehensively analyzed audio representation learning for non-speech audio tasks. In this paper, we propose a self-supervised audio representation learning method and apply it to a variety of downstream non-speech audio tasks. We combine the well-known wav2vec 2.0 framework, which has shown success in self-supervised learning for speech tasks, with parameter-efficient conformer architectures. Our self-supervised pre-training can reduce the need for labeled data by two-thirds. On the AudioSet benchmark, we achieve a mean average precision (mAP) score of 0.415, which is a new state-of-the-art on this dataset through audio-only self-supervised learning. Our fine-tuned conformers also surpass or match the performance of previous systems pre-trained in a supervised way on several downstream tasks. We further discuss the important design considerations for both pre-training and fine-tuning.