# Use of speaker recognition approaches for learning and evaluating embedding representations of musical instrument sounds

arXiv: https://arxiv.org/abs/2107.11506 · HTML: https://mlsanity.com/paper/2107.11506/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2021.

## Built on by (most cited first, 5 of 5)

- [Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio](https://mlsanity.com/paper/2211.07951/) (2022)
- [Towards Robust Few-shot Class Incremental Learning in Audio Classification using Contrastive Representation](https://mlsanity.com/paper/2407.19265/) (2024)
- [Pitch-Conditioned Instrument Sound Synthesis From an Interactive Timbre Latent Space](https://mlsanity.com/paper/2510.04339/) (2025)
- [Few-shot Class-variable Incremental Audio Classification via Prototype Adaptation and Pseudo Class-variable Training](https://mlsanity.com/paper/2606.08898/) (2026)
- [CPR: Combining global composing, local performing and full-sequence refining in piano rendering with continuous autoregressive modelling](https://mlsanity.com/paper/2609.18216/) (2026)

## Abstract (arXiv)

Constructing an embedding space for musical instrument sounds that can meaningfully represent new and unseen instruments is important for downstream music generation tasks such as multi-instrument synthesis and timbre transfer. The framework of Automatic Speaker Verification (ASV) provides us with architectures and evaluation methodologies for verifying the identities of unseen speakers, and these can be repurposed for the task of learning and evaluating a musical instrument sound embedding space that can support unseen instruments. Borrowing from state-of-the-art ASV techniques, we construct a musical instrument recognition model that uses a SincNet front-end, a ResNet architecture, and an angular softmax objective function. Experiments on the NSynth and RWC datasets show our model's effectiveness in terms of equal error rate (EER) for unseen instruments, and ablation studies show the importance of data augmentation and the angular softmax objective. Experiments also show the benefit of using a CQT-based filterbank for initializing SincNet over a Mel filterbank initialization. Further complementary analysis of the learned embedding space is conducted with t-SNE visualizations and probing classification tasks, which show that including instrument family labels as a multi-task learning target can help to regularize the embedding space and incorporate useful structure, and that meaningful information such as playing style, which was not included during training, is contained in the embeddings of unseen instruments.