# A Foundation Model for Music Informatics

arXiv: https://arxiv.org/abs/2311.03318 · HTML: https://mlsanity.com/paper/2311.03318/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2023.

## Built on by (most cited first, 30 of 46)

- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization](https://mlsanity.com/paper/2501.01108/) (2025)
- [Do Music Generation Models Encode Music Theory?](https://mlsanity.com/paper/2410.00872/) (2024)
- [SongCreator: Lyrics-based Universal Song Generation](https://mlsanity.com/paper/2409.06029/) (2024)
- [OpenMU: Your Swiss Army Knife for Music Understanding](https://mlsanity.com/paper/2410.15573/) (2024)
- [SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing](https://mlsanity.com/paper/2601.09385/) (2026)
- [LC-Protonets: Multi-Label Few-Shot Learning for World Music Audio Tagging](https://mlsanity.com/paper/2409.11264/) (2024)
- [Layer-wise Investigation of Large-Scale Self-Supervised Music Representation Models](https://mlsanity.com/paper/2505.16306/) (2025)
- [Myna: Masking-Based Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2502.12511/) (2025)
- [Universal Music Representations? Evaluating Foundation Models on World Music Corpora](https://mlsanity.com/paper/2506.17055/) (2025)
- [From Discord to Harmony: Decomposed Consonance-based Training for Improved Audio Chord Estimation](https://mlsanity.com/paper/2509.01588/) (2025)
- [Embedding Compression for Teacher-to-Student Knowledge Transfer](https://mlsanity.com/paper/2402.06761/) (2024)
- [ChordSync: Conformer-Based Alignment of Chord Annotations to Music Audio](https://mlsanity.com/paper/2408.00674/) (2024)
- [CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning](https://mlsanity.com/paper/2506.17818/) (2025)
- [Emergent musical properties of a transformer under contrastive self-supervised learning](https://mlsanity.com/paper/2506.23873/) (2025)
- [V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generation](https://mlsanity.com/paper/2603.11042/) (2026)
- [MuScriptor: An Open Model for Multi-Instrument Music Transcription](https://mlsanity.com/paper/2607.08168/) (2026)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [Keep what you need : extracting efficient subnetworks from large audio representation models](https://mlsanity.com/paper/2502.12925/) (2025)
- [Do Foundational Audio Encoders Understand Music Structure?](https://mlsanity.com/paper/2512.17209/) (2025)
- [Evaluating Disentangled Representations for Controllable Music Generation](https://mlsanity.com/paper/2602.10058/) (2026)
- [USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding](https://mlsanity.com/paper/2606.06444/) (2026)
- [Benchmarking Sub-Genre Classification For Mainstage Dance Music](https://mlsanity.com/paper/2409.06690/) (2024)
- [LAV: Audio-Driven Dynamic Visual Generation with Neural Compression and StyleGAN2](https://mlsanity.com/paper/2505.10101/) (2025)
- [Semantic-Aware Interpretable Multimodal Music Auto-Tagging](https://mlsanity.com/paper/2505.17233/) (2025)
- [Improving BERT for Symbolic Music Understanding Using Token Denoising and Pianoroll Prediction](https://mlsanity.com/paper/2507.04776/) (2025)
- [A Study on the Data Distribution Gap in Music Emotion Recognition](https://mlsanity.com/paper/2510.04688/) (2025)
- [Transcribing Rhythmic Patterns of the Guitar Track in Polyphonic Music](https://mlsanity.com/paper/2510.05756/) (2025)
- [Hear: Hierarchically Enhanced Aesthetic Representations For Multidimensional Music Evaluation](https://mlsanity.com/paper/2511.18869/) (2025)

## Abstract (arXiv)

This paper investigates foundation models tailored for music informatics, a domain currently challenged by the scarcity of labeled data and generalization issues. To this end, we conduct an in-depth comparative study among various foundation model variants, examining key determinants such as model architectures, tokenization methods, temporal resolution, data, and model scalability. This research aims to bridge the existing knowledge gap by elucidating how these individual factors contribute to the success of foundation models in music informatics. Employing a careful evaluation framework, we assess the performance of these models across diverse downstream tasks in music information retrieval, with a particular focus on token-level and sequence-level classification. Our results reveal that our model demonstrates robust performance, surpassing existing models in specific key metrics. These findings contribute to the understanding of self-supervised learning in music informatics and pave the way for developing more effective and versatile foundation models in the field. A pretrained version of our model is publicly available to foster reproducibility and future research.