# Leveraging Pre-Trained Autoencoders for Interpretable Prototype Learning of Music Audio

arXiv: https://arxiv.org/abs/2402.09318 · HTML: https://mlsanity.com/paper/2402.09318/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Builds on (most cited first, 15 of 15)

- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Toward Interpretable Music Tagging with Self-Attention](https://mlsanity.com/paper/1906.04972/) (2019)
- [Deep convolutional networks on the pitch spiral for musical instrument recognition](https://mlsanity.com/paper/1605.06644/) (2016)
- [Sample-level CNN Architectures for Music Auto-tagging Using Raw Waveforms](https://mlsanity.com/paper/1710.10451/) (2017)
- [Joint Time-Frequency Scattering](https://mlsanity.com/paper/1807.08869/) (2018)
- [EnCodecMAE: Leveraging neural codecs for universal audio representation learning](https://mlsanity.com/paper/2309.07391/) (2023)
- [A Model You Can Hear: Audio Identification with Playable Prototypes](https://mlsanity.com/paper/2208.03311/) (2022)
- [Efficient Supervised Training of Audio Transformers for Music Representation Learning](https://mlsanity.com/paper/2309.16418/) (2023)
- [Leveraging Knowledge Bases And Parallel Annotations For Music Genre Translation](https://mlsanity.com/paper/1907.08698/) (2019)
- [The "Horse'' Inside: Seeking Causes Behind the Behaviours of Music Content Analysis Systems](https://mlsanity.com/paper/1606.03044/) (2016)

## Built on by (most cited first, 5 of 5)

- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Music2Latent: Consistency Autoencoders for Latent Audio Compression](https://mlsanity.com/paper/2408.06500/) (2024)
- [Listenable Maps for Zero-Shot Audio Classifiers](https://mlsanity.com/paper/2405.17615/) (2024)
- [MusicLIME: Explainable Multimodal Music Understanding](https://mlsanity.com/paper/2409.10496/) (2024)
- [Semantic-Aware Interpretable Multimodal Music Auto-Tagging](https://mlsanity.com/paper/2505.17233/) (2025)

## Abstract (arXiv)

We present PECMAE, an interpretable model for music audio classification based on prototype learning. Our model is based on a previous method, APNet, which jointly learns an autoencoder and a prototypical network. Instead, we propose to decouple both training processes. This enables us to leverage existing self-supervised autoencoders pre-trained on much larger data (EnCodecMAE), providing representations with better generalization. APNet allows prototypes' reconstruction to waveforms for interpretability relying on the nearest training data samples. In contrast, we explore using a diffusion decoder that allows reconstruction without such dependency. We evaluate our method on datasets for music instrument classification (Medley-Solos-DB) and genre recognition (GTZAN and a larger in-house dataset), the latter being a more challenging task not addressed with prototypical networks before. We find that the prototype-based models preserve most of the performance achieved with the autoencoder embeddings, while the sonification of prototypes benefits understanding the behavior of the classifier.