Do Foundational Audio Encoders Understand Music Structure?
Not on the map. Why?
Abstract (arXiv)
In music information retrieval (MIR) research, the use of pretrained foundational audio encoders (FAEs) has recently become a trend. FAEs pretrained on large amounts of music and audio data have been shown to improve performance on MIR tasks such as music tagging and automatic music transcription. However, their use for music structure analysis (MSA) remains underexplored: only a small subset of FAEs has been examined for MSA, and the impact of factors such as learning methods, training data, and model context length on MSA performance remains unclear. In this study, we conduct comprehensive experiments on 11 types of FAEs to investigate how these factors affect MSA performance. Our results demonstrate that FAEs using self-supervised learning with masked language modeling on music data are particularly effective for MSA. These findings pave the way for future research in FAE and MSA.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 19 Dec 2025.
Builds on
29 of 45 references.
References · 29
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
- 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
- 2022High Fidelity Neural Audio Compressioncited by 673
- 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
- 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
- 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
- 2022Masked Autoencoders that Listencited by 271
- 2021Efficient Training of Audio Transformers with Patchoutcited by 222
- 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
- 2016FMA: A Dataset For Music Analysiscited by 205
- 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
- 2023Adapting Frechet Audio Distance for Generative Music Evaluationcited by 92
17 more
- 2024Long-form music generation with latent diffusioncited by 88
- 2023A Foundation Model for Music Informaticscited by 45
- 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
- 2023All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audiocited by 27
- 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speechcited by 25
- 2022Melody transcription via generative pre-trainingcited by 23
- 2025Aligning Text-to-Music Evaluation with Human Preferencescited by 20
- 2022To catch a chorus, verse, intro, or anything else: Analyzing a song with structural functionscited by 10
- 2021Supervised Metric Learning for Music Structure Featurescited by 6
- 2024Music Foundation Model as Generic Booster for Music Downstream Taskscited by 6
- 2023Extending Audio Masked Autoencoders Toward Audio Restorationcited by 5
- 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
- 2019Enhanced Hierarchical Music Structure Annotations via Feature Level Similarity Fusioncited by 4
- 2023Self-Similarity-Based and Novelty-based loss for music structure analysiscited by 3
- 2022SSM-Net: feature learning for Music Structure Analysis using a Self-Similarity-Matrix based losscited by 2
- 2021Supervised Chorus Detection for Popular Music Using Convolutional Neural Network and Multi-task Learningcited by 1
- 2025Temporal Adaptation of Pre-trained Foundation Models for Music Structure Analysiscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.