# Learning Features of Music from Scratch

arXiv: https://arxiv.org/abs/1611.09827 · HTML: https://mlsanity.com/paper/1611.09827/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2016.

## Built on by (most cited first, 30 of 58)

- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Ola: Pushing the Frontiers of Omni-Modal Language Model](https://mlsanity.com/paper/2502.04328/) (2025)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Calibrated Multiple-Output Quantile Regression with Representation Learning](https://mlsanity.com/paper/2110.00816/) (2021)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [Melody transcription via generative pre-training](https://mlsanity.com/paper/2212.01884/) (2022)
- [Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?](https://mlsanity.com/paper/2402.12025/) (2024)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [Opening the Black Box of wav2vec Feature Encoder](https://mlsanity.com/paper/2210.15386/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Building Blocks for a Complex-Valued Transformer Architecture](https://mlsanity.com/paper/2306.09827/) (2023)
- [A Data-Driven Analysis of Robust Automatic Piano Transcription](https://mlsanity.com/paper/2402.01424/) (2024)
- [High Resolution Guitar Transcription via Domain Adaptation](https://mlsanity.com/paper/2402.15258/) (2024)
- [Omnizart: A General Toolbox for Automatic Music Transcription](https://mlsanity.com/paper/2106.00497/) (2021)
- [YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation](https://mlsanity.com/paper/2407.04822/) (2024)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [A2SB: Audio-to-Audio Schrodinger Bridges](https://mlsanity.com/paper/2501.11311/) (2025)
- [The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling](https://mlsanity.com/paper/2209.14458/) (2022)
- [Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling](https://mlsanity.com/paper/2504.15071/) (2025)
- [WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning](https://mlsanity.com/paper/2509.04744/) (2025)
- [OpenMU: Your Swiss Army Knife for Music Understanding](https://mlsanity.com/paper/2410.15573/) (2024)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Music Foundation Model as Generic Booster for Music Downstream Tasks](https://mlsanity.com/paper/2411.01135/) (2024)
- [Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio](https://mlsanity.com/paper/2505.12863/) (2025)
- [Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music](https://mlsanity.com/paper/2107.05916/) (2021)
- [Soft Dynamic Time Warping for Multi-Pitch Estimation and Beyond](https://mlsanity.com/paper/2304.05032/) (2023)
- [PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations](https://mlsanity.com/paper/2410.02060/) (2024)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)

## Abstract (arXiv)

This paper introduces a new large-scale music dataset, MusicNet, to serve as a source of supervision and evaluation of machine learning methods for music research. MusicNet consists of hundreds of freely-licensed classical music recordings by 10 composers, written for 11 instruments, together with instrument/note annotations resulting in over 1 million temporal labels on 34 hours of chamber music performances under various studio and microphone conditions. The paper defines a multi-label classification task to predict notes in musical recordings, along with an evaluation protocol, and benchmarks several machine learning architectures for this task: i) learning from spectrogram features; ii) end-to-end learning with a neural net; iii) end-to-end learning with a convolutional neural net. These experiments show that end-to-end models trained for note prediction learn frequency selective filters as a low-level representation of audio.