# Toward Interpretable Music Tagging with Self-Attention

arXiv: https://arxiv.org/abs/1906.04972 · HTML: https://mlsanity.com/paper/1906.04972/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 30 of 30)

- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMF](https://mlsanity.com/paper/2202.11479/) (2022)
- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [SpecTNT: a Time-Frequency Transformer for Music Audio](https://mlsanity.com/paper/2110.09127/) (2021)
- [Listenable Maps for Audio Classifiers](https://mlsanity.com/paper/2403.13086/) (2024)
- [The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling](https://mlsanity.com/paper/2209.14458/) (2022)
- [AudioProtoPNet: An interpretable deep learning model for bird sound classification](https://mlsanity.com/paper/2404.10420/) (2024)
- [Tracing Back Music Emotion Predictions to Sound Sources and Intuitive Perceptual Qualities](https://mlsanity.com/paper/2106.07787/) (2021)
- [Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features](https://mlsanity.com/paper/2309.07733/) (2023)
- [Perceptual Musical Features for Interpretable Audio Tagging](https://mlsanity.com/paper/2312.11234/) (2023)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Interpreting Song Lyrics with an Audio-Informed Pre-trained Language Model](https://mlsanity.com/paper/2208.11671/) (2022)
- [Tackling Interpretability in Audio Classification Networks with Non-negative Matrix Factorization](https://mlsanity.com/paper/2305.07132/) (2023)
- [Learning Music Sequence Representation from Text Supervision](https://mlsanity.com/paper/2305.19602/) (2023)
- [Audio Embeddings as Teachers for Music Classification](https://mlsanity.com/paper/2306.17424/) (2023)
- [Leveraging Pre-Trained Autoencoders for Interpretable Prototype Learning of Music Audio](https://mlsanity.com/paper/2402.09318/) (2024)
- [Transformation of audio embeddings into interpretable, concept-based representations](https://mlsanity.com/paper/2504.14076/) (2025)
- [Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Features](https://mlsanity.com/paper/2110.08862/) (2021)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [MusicLIME: Explainable Multimodal Music Understanding](https://mlsanity.com/paper/2409.10496/) (2024)
- [One Wave To Explain Them All: A Unifying Perspective On Feature Attribution](https://mlsanity.com/paper/2410.01482/) (2024)
- [ALCAP: Alignment-Augmented Music Captioner](https://mlsanity.com/paper/2212.10901/) (2022)
- [TransCORALNet: A Two-Stream Transformer CORAL Networks for Supply Chain Credit Assessment Cold Start](https://mlsanity.com/paper/2311.18749/) (2023)
- [Music Auto-Tagging with Robust Music Representation Learned via Domain Adversarial Training](https://mlsanity.com/paper/2401.15323/) (2024)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [MuSLCAT: Multi-Scale Multi-Level Convolutional Attention Transformer for Discriminative Music Modeling on Raw Waveforms](https://mlsanity.com/paper/2104.02309/) (2021)
- [Semantic-Aware Interpretable Multimodal Music Auto-Tagging](https://mlsanity.com/paper/2505.17233/) (2025)
- [Are Inherently Interpretable Models More Robust? A Study In Music Emotion Recognition](https://mlsanity.com/paper/2508.03780/) (2025)

## Abstract (arXiv)

Self-attention is an attention mechanism that learns a representation by relating different positions in the sequence. The transformer, which is a sequence model solely based on self-attention, and its variants achieved state-of-the-art results in many natural language processing tasks. Since music composes its semantics based on the relations between components in sparse positions, adopting the self-attention mechanism to solve music information retrieval (MIR) problems can be beneficial. Hence, we propose a self-attention based deep sequence model for music tagging. The proposed architecture consists of shallow convolutional layers followed by stacked Transformer encoders. Compared to conventional approaches using fully convolutional or recurrent neural networks, our model is more interpretable while reporting competitive results. We validate the performance of our model with the MagnaTagATune and the Million Song Dataset. In addition, we demonstrate the interpretability of the proposed architecture with a heat map visualization.