# Semantic-Aware Interpretable Multimodal Music Auto-Tagging

arXiv: https://arxiv.org/abs/2505.17233 · HTML: https://mlsanity.com/paper/2505.17233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 16 of 16)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [A Foundation Model for Music Informatics](https://mlsanity.com/paper/2311.03318/) (2023)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Toward Interpretable Music Tagging with Self-Attention](https://mlsanity.com/paper/1906.04972/) (2019)
- [Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms](https://mlsanity.com/paper/1703.01789/) (2017)
- [Omnizart: A General Toolbox for Automatic Music Transcription](https://mlsanity.com/paper/2106.00497/) (2021)
- [Concept-Based Techniques for "Musicologist-friendly" Explanations in a Deep Music Classifier](https://mlsanity.com/paper/2208.12485/) (2022)
- [Towards Explainable Music Emotion Recognition: The Route via Mid-level Features](https://mlsanity.com/paper/1907.03572/) (2019)
- [A data-driven approach to mid-level perceptual musical feature modeling](https://mlsanity.com/paper/1806.04903/) (2018)
- [Perceptual Musical Features for Interpretable Audio Tagging](https://mlsanity.com/paper/2312.11234/) (2023)
- [A Model You Can Hear: Audio Identification with Playable Prototypes](https://mlsanity.com/paper/2208.03311/) (2022)
- [Leveraging Pre-Trained Autoencoders for Interpretable Prototype Learning of Music Audio](https://mlsanity.com/paper/2402.09318/) (2024)
- [MusicLIME: Explainable Multimodal Music Understanding](https://mlsanity.com/paper/2409.10496/) (2024)

## Abstract (arXiv)

Music auto-tagging is essential for organizing and discovering music in extensive digital libraries. While foundation models achieve exceptional performance in this domain, their outputs often lack interpretability, limiting trust and usability for researchers and end-users alike. In this work, we present an interpretable framework for music auto-tagging that leverages groups of musically meaningful multimodal features, derived from signal processing, deep learning, ontology engineering, and natural language processing. To enhance interpretability, we cluster features semantically and employ an expectation maximization algorithm, assigning distinct weights to each group based on its contribution to the tagging process. Our method achieves competitive tagging performance while offering a deeper understanding of the decision-making process, paving the way for more transparent and user-centric music tagging systems.