# Unsupervised Musical Object Discovery from Audio

arXiv: https://arxiv.org/abs/2311.07534 · HTML: https://mlsanity.com/paper/2311.07534/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [Tagger: Deep Unsupervised Perceptual Grouping](https://mlsanity.com/paper/1606.06724/) (2016)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation](https://mlsanity.com/paper/2207.00787/) (2022)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning](https://mlsanity.com/paper/2212.02508/) (2022)
- [R-SQAIR: Relational Sequential Attend, Infer, Repeat](https://mlsanity.com/paper/1910.05231/) (2019)
- [Unsupervised Learning of Temporal Abstractions with Slot-based Transformers](https://mlsanity.com/paper/2203.13573/) (2022)
- [Learning music audio representations via weak language supervision](https://mlsanity.com/paper/2112.04214/) (2021)
- [Modularity through Attention: Efficient Training and Transfer of Language-Conditioned Policies for Robot Manipulation](https://mlsanity.com/paper/2212.04573/) (2022)
- [Representation Learning of Music Using Artist Labels](https://mlsanity.com/paper/1710.06648/) (2017)
- [Towards Proper Contrastive Self-supervised Learning Strategies For Music Audio Representation](https://mlsanity.com/paper/2207.04471/) (2022)
- [AudioSlots: A slot-centric generative model for audio separation](https://mlsanity.com/paper/2305.05591/) (2023)
- [Learning Music Sequence Representation from Text Supervision](https://mlsanity.com/paper/2305.19602/) (2023)
- [jazznet: A Dataset of Fundamental Piano Patterns for Music Audio Machine Learning Research](https://mlsanity.com/paper/2302.08632/) (2023)

## Built on by (most cited first, 2 of 2)

- [Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery](https://mlsanity.com/paper/2405.17283/) (2024)
- [DisMix: Disentangling Mixtures of Musical Instruments for Source-level Pitch and Timbre Manipulation](https://mlsanity.com/paper/2408.10807/) (2024)

## Abstract (arXiv)

Current object-centric learning models such as the popular SlotAttention architecture allow for unsupervised visual scene decomposition. Our novel MusicSlots method adapts SlotAttention to the audio domain, to achieve unsupervised music decomposition. Since concepts of opacity and occlusion in vision have no auditory analogues, the softmax normalization of alpha masks in the decoders of visual object-centric models is not well-suited for decomposing audio objects. MusicSlots overcomes this problem. We introduce a spectrogram-based multi-object music dataset tailored to evaluate object-centric learning on western tonal music. MusicSlots achieves good performance on unsupervised note discovery and outperforms several established baselines on supervised note property prediction tasks.