# Frame-level Instrument Recognition by Timbre and Pitch

arXiv: https://arxiv.org/abs/1806.09587 · HTML: https://mlsanity.com/paper/1806.09587/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2018.

## Built on by (most cited first, 7 of 7)

- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [Leveraging Hierarchical Structures for Few-Shot Musical Instrument Recognition](https://mlsanity.com/paper/2107.07029/) (2021)
- [Frame-Level Multi-Label Playing Technique Detection Using Multi-Scale Network and Self-Attention Mechanism](https://mlsanity.com/paper/2303.13272/) (2023)
- [Music Instrument Classification Reprogrammed](https://mlsanity.com/paper/2211.08379/) (2022)
- [Self-refining of Pseudo Labels for Music Source Separation with Noisy Labeled Data](https://mlsanity.com/paper/2307.12576/) (2023)
- [Timbre Classification of Musical Instruments with a Deep Learning Multi-Head Attention-Based Model](https://mlsanity.com/paper/2107.06231/) (2021)
- [Pitch-Informed Instrument Assignment Using a Deep Convolutional Network with Multiple Kernel Shapes](https://mlsanity.com/paper/2107.13617/) (2021)

## Abstract (arXiv)

Instrument recognition is a fundamental task in music information retrieval, yet little has been done to predict the presence of instruments in multi-instrument music for each time frame. This task is important for not only automatic transcription but also many retrieval problems. In this paper, we use the newly released MusicNet dataset to study this front, by building and evaluating a convolutional neural network for making frame-level instrument prediction. We consider it as a multi-label classification problem for each frame and use frame-level annotations as the supervisory signal in training the network. Moreover, we experiment with different ways to incorporate pitch information to our model, with the premise that doing so informs the model the notes that are active per frame, and also encourages the model to learn relative rates of energy buildup in the harmonic partials of different instruments. Experiments show salient performance improvement over baseline methods. We also report an analysis probing how pitch information helps the instrument prediction task. Code and experiment details can be found at urlhttps://biboamy.github.io/instrument-recognition/.