# Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio

arXiv: https://arxiv.org/abs/2211.07951 · HTML: https://mlsanity.com/paper/2211.07951/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 16 of 16)

- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity](https://mlsanity.com/paper/1909.08494/) (2019)
- [Deep convolutional networks on the pitch spiral for musical instrument recognition](https://mlsanity.com/paper/1605.06644/) (2016)
- [Leveraging Hierarchical Structures for Few-Shot Musical Instrument Recognition](https://mlsanity.com/paper/2107.07029/) (2021)
- [Deep convolutional neural networks for predominant instrument recognition in polyphonic music](https://mlsanity.com/paper/1605.09507/) (2016)
- [Many-Speakers Single Channel Speech Separation with Optimal Permutation Training](https://mlsanity.com/paper/2104.08955/) (2021)
- [Few-Shot Drum Transcription in Polyphonic Music](https://mlsanity.com/paper/2008.02791/) (2020)
- [Augmentation Methods on Monophonic Audio for Instrument Classification in Polyphonic Music](https://mlsanity.com/paper/1911.12505/) (2019)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Use of speaker recognition approaches for learning and evaluating embedding representations of musical instrument sounds](https://mlsanity.com/paper/2107.11506/) (2021)
- [Automatic Instrument Recognition in Polyphonic Music Using Convolutional Neural Networks](https://mlsanity.com/paper/1511.05520/) (2015)
- [Similarity measures for vocal-based drum sample retrieval using deep convolutional auto-encoders](https://mlsanity.com/paper/1802.05178/) (2018)
- [Deep Convolutional and Recurrent Networks for Polyphonic Instrument Classification from Monophonic Raw Audio Waveforms](https://mlsanity.com/paper/2102.06930/) (2021)

## Built on by (most cited first, 4 of 4)

- [TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument](https://mlsanity.com/paper/2502.08939/) (2025)
- [Annotation-free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion](https://mlsanity.com/paper/2312.10402/) (2023)
- [Self-refining of Pseudo Labels for Music Source Separation with Noisy Labeled Data](https://mlsanity.com/paper/2307.12576/) (2023)
- [Contrastive timbre representations for musical instrument and synthesizer retrieval](https://mlsanity.com/paper/2509.13285/) (2025)

## Abstract (arXiv)

As digital music production has become mainstream, the selection of appropriate virtual instruments plays a crucial role in determining the quality of music. To search the musical instrument samples or virtual instruments that make one's desired sound, music producers use their ears to listen and compare each instrument sample in their collection, which is time-consuming and inefficient. In this paper, we call this task as Musical Instrument Retrieval and propose a method for retrieving desired musical instruments using reference music mixture as a query. The proposed model consists of the Single-Instrument Encoder and the Multi-Instrument Encoder, both based on convolutional neural networks. The Single-Instrument Encoder is trained to classify the instruments used in single-track audio, and we take its penultimate layer's activation as the instrument embedding. The Multi-Instrument Encoder is trained to estimate multiple instrument embeddings using the instrument embeddings computed by the Single-Instrument Encoder as a set of target embeddings. For more generalized training and realistic evaluation, we also propose a new dataset called Nlakh. Experimental results showed that the Single-Instrument Encoder was able to learn the mapping from the audio signal of unseen instruments to the instrument embedding space and the Multi-Instrument Encoder was able to extract multiple embeddings from the mixture of music and retrieve the desired instruments successfully. The code used for the experiment and audio samples are available at: https://github.com/minju0821/musical_instrument_retrieval