mlsanityData to 14 Oct 2026

MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

arXiv 2306.00107 ↗First posted 31 May 2023cs.SD cs.AI cs.CLcited by 181

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored. This is partially due to the distinctive challenges associated with modelling musical knowledge, particularly tonal and pitched characteristics of music. To address this research gap, we propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. In our exploration, we identified an effective combination of teacher models, which outperforms conventional speech and audio approaches in terms of performance. This combination includes an acoustic teacher based on Residual Vector Quantisation - Variational AutoEncoder (RVQ-VAE) and a musical teacher based on the Constant-Q Transform (CQT). Furthermore, we explore a wide range of settings to overcome the instability in acoustic language model pre-training, which allows our designed paradigm to scale from 95M to 330M parameters. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores.

Key facts. Cited by 181 papers in the map (data through 14 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

38 of 64 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2022High Fidelity Neural Audio Compressioncited by 687
26 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2020Jukebox: A Generative Model for Musiccited by 542
  4. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  5. 2019Deep Learning for Symbolic Mathematicscited by 270
  6. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  7. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  8. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  9. 2022Hybrid Transformers for Music Source Separationcited by 134
  10. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 87
  11. 2021Contrastive Learning of Musical Representationscited by 82
  12. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  13. 2016madmom: a new Python Audio and Music Signal Processing Librarycited by 64
  14. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 64
  15. 2021Towards Learning Universal Audio Representationscited by 51
  16. 2019musicnn: Pre-trained convolutional neural networks for music audio taggingcited by 50
  17. 2021Music Demixing Challenge 2021cited by 38
  18. 2017Transfer learning for music classification and regression taskscited by 37
  19. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 35
  20. 2022MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learningcited by 22
  21. 2015Deep Learning and Music Adversariescited by 20
  22. 2021The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtrackscited by 19
  23. 2017End-to-End Musical Key Estimation Using a Convolutional Neural Networkcited by 13
  24. 2021BeatNet: CRNN and Particle Filtering for Online Joint Beat Downbeat and Meter Trackingcited by 13
  25. 2022Deformable CNN and Imbalance-Aware Feature Learning for Singing Technique Classificationcited by 8
  26. 2021Transfer Learning with Jukebox for Music Source Separationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 41; 2025: 60; 2026: 67.

181 papers.

Sort:

By area:

  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  3. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  4. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  5. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
  6. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  7. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  8. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  9. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  10. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  11. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 68
  12. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 61
38 more
  1. 2023MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Responsecited by 45
  2. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
  3. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  4. 2024Foundation Models for Music: A Surveycited by 37
  5. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  6. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 33
  7. 2024Music2Latent: Consistency Autoencoders for Latent Audio Compressioncited by 33
  8. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  9. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  10. 2025SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generationcited by 29
  11. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  12. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  13. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinementcited by 23
  14. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  15. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 17
  16. 2024MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Musiccited by 16
  17. 2023LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPTcited by 15
  18. 2025JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadatacited by 15
  19. 2024Do Music Generation Models Encode Music Theory?cited by 13
  20. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  21. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  22. 2024The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Modelscited by 10
  23. 2024OpenMU: Your Swiss Army Knife for Music Understandingcited by 9
  24. 2025Scaling Self-Supervised Representation Learning for Symbolic Piano Performancecited by 9
  25. 2025Align Your Rhythm: Generating Highly Aligned Dance Poses with Gating-Enhanced Rhythm-Aware Feature Representationcited by 8
  26. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 8
  27. 2024Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challengescited by 7
  28. 2025Towards Unified Music Emotion Recognition across Dimensional and Categorical Modelscited by 7
  29. 2025Detect All-Type Deepfake Audio: Wavelet Prompt Tuning for Enhanced Auditory Perceptioncited by 7
  30. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  31. 2024SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognitioncited by 6
  32. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  33. 2025MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learningcited by 6
  34. 2026Interpretable All-Type Audio Deepfake Detection with Audio LLMs via Frequency-Time Reinforcement Learningcited by 6
  35. 2026MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluationcited by 6
  36. 2023Joint Music and Language Attention Models for Zero-shot Music Taggingcited by 5
  37. 2024Toward Fully Self-Supervised Multi-Pitch Estimationcited by 5
  38. 2024FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generationcited by 5

50 most cited of 181 All 181 (JSON)