mlsanityData to 14 Oct 2026

On the Effectiveness of Speech Self-supervised Learning for Music

arXiv 2307.05161 ↗First posted 11 Jul 2023cs.SD cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has shown promising results in various speech and natural language processing applications. However, its efficacy in music information retrieval (MIR) still remains largely unexplored. While previous SSL models pre-trained on music recordings may have been mostly closed-sourced, recent speech models such as wav2vec2.0 have shown promise in music modelling. Nevertheless, research exploring the effectiveness of applying speech SSL models to music recordings has been limited. We explore the music adaption of SSL with two distinctive speech-related models, data2vec1.0 and Hubert, and refer to them as music2vec and musicHuBERT, respectively. We train 12 SSL models with 95M parameters under various pre-training configurations and systematically evaluate the MIR task performances with 13 different MIR tasks. Our findings suggest that training with music data can generally improve performance on MIR tasks, even when models are trained using paradigms designed for speech. However, we identify the limitations of such existing speech-oriented designs, especially in modelling polyphonic information. Based on the experimental results, empirical suggestions are also given for designing future musical SSL strategies and paradigms.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

28 of 50 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  9. 2020Jukebox: A Generative Model for Musiccited by 542
  10. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  11. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  12. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
16 more
  1. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 130
  2. 2021Contrastive Learning of Musical Representationscited by 82
  3. 2020How Useful is Self-Supervised Pretraining for Visual Tasks?cited by 80
  4. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  5. 2016madmom: a new Python Audio and Music Signal Processing Librarycited by 64
  6. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 64
  7. 2021Towards Learning Universal Audio Representationscited by 51
  8. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 35
  9. 2022MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learningcited by 22
  10. 2015Deep Learning and Music Adversariescited by 20
  11. 2021Multi-Task Self-Supervised Pre-Training for Music Classificationcited by 18
  12. 2017End-to-End Musical Key Estimation Using a Convolutional Neural Networkcited by 13
  13. 2021BeatNet: CRNN and Particle Filtering for Online Joint Beat Downbeat and Meter Trackingcited by 13
  14. 2022Contrastive Learning with Positive-Negative Frame Mask for Music Representationcited by 10
  15. 2022Deformable CNN and Imbalance-Aware Feature Learning for Singing Technique Classificationcited by 8
  16. 2022Learning Music Representations with wav2vec 2.0cited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 2.

5 papers.

  1. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 68
  2. 2024Foundation Models for Music: A Surveycited by 37
  3. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 33
  4. 2025Segment Transformer: AI-Generated Music Detection via Music Structural Analysiscited by 1
  5. 2023Audio Contrastive-based Fine-tuning: Decoupling Representation Learning and Classificationcited by 0