Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods
Not on the map. Why?
Abstract (arXiv)
Current audio pre-training seeks to learn unified representations for broad audio understanding tasks, but it remains fragmented and is fundamentally bottlenecked by its reliance on weak, noisy, and scale-limited labels. Drawing lessons from vision's foundational pre-training blueprint, we argue that the audio field must first establish its own large-scale, strong supervision framework. We introduce a new data-centric pipeline that leverages a high-fidelity captioner to create SOTA-quality captions and the first Unified Tag System (UTS) that bridges speech, music, and environmental sounds. We then conduct a systematic comparative study of different pre-training objectives on these strong source data. Our experiments suggest that data quality and coverage are the primary drivers of performance, while the choice of objective dictates downstream task specialization.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 69 of its 93 references that are in the map. First posted on arXiv 26 Mar 2026.
Builds on
69 of 93 references.
References · 69
- 2015Deep Residual Learning for Image Recognitioncited by 51,950
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
38 more
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,625
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,366
- 2018VoxCeleb2: Deep Speaker Recognitioncited by 864
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 721
- 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 713
- 2016CNN Architectures for Large-Scale Audio Classificationcited by 702
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 657
- 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 644
- 2021AST: Audio Spectrogram Transformercited by 639
- 2023DataComp: In search of the next generation of multimodal datasetscited by 639
- 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 637
- 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 612
- 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
- 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 567
- 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
- 2025Qwen3-Omni Technical Reportcited by 464
- 2017Non-Autoregressive Neural Machine Translationcited by 462
- 2023MusicLM: Generating Music From Textcited by 445
- 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 384
- 2019Clotho: An Audio Captioning Datasetcited by 356
- 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 340
- 2022Masked Autoencoders that Listencited by 272
- 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
- 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
- 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 217
- 2023Improving CLIP Training with Language Rewritescited by 216
- 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 213
- 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 211
- 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 177
- 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 175
- 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 168
- 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
- 2023Image Captioners Are Scalable Vision Learners Toocited by 89
- 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 86
50 most cited of 69 All 69 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.