# Spiking Music: Audio Compression with Event Based Auto-encoders

arXiv: https://arxiv.org/abs/2402.01571 · HTML: https://mlsanity.com/paper/2402.01571/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 30 of 34)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Surrogate Gradient Learning in Spiking Neural Networks](https://mlsanity.com/paper/1901.09948/) (2019)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Long short-term memory and learning-to-learn in networks of spiking neurons](https://mlsanity.com/paper/1803.09574/) (2018)
- [The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks](https://mlsanity.com/paper/1910.07407/) (2019)
- [Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing](https://mlsanity.com/paper/1603.08270/) (2016)
- [The challenge of realistic music generation: modelling raw audio at scale](https://mlsanity.com/paper/1806.10474/) (2018)
- [A Unified, Scalable Framework for Neural Population Decoding](https://mlsanity.com/paper/2310.16046/) (2023)
- [Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks](https://mlsanity.com/paper/2009.08378/) (2020)
- [Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder](https://mlsanity.com/paper/1910.06464/) (2019)
- [SEANet: A Multi-modal Speech Enhancement Network](https://mlsanity.com/paper/2009.02095/) (2020)
- [Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder](https://mlsanity.com/paper/1802.02305/) (2018)
- [Efficient recurrent architectures through activity sparsity and sparse back-propagation through time](https://mlsanity.com/paper/2206.06178/) (2022)
- [Variable-rate discrete representation learning](https://mlsanity.com/paper/2103.06089/) (2021)
- [High-performance deep spiking neural networks with 0.3 spikes per neuron](https://mlsanity.com/paper/2306.08744/) (2023)
- [Fitting New Speakers Based on a Short Untranscribed Sample](https://mlsanity.com/paper/1802.06984/) (2018)
- [Single channel voice separation for unknown number of speakers under reverberant and noisy settings](https://mlsanity.com/paper/2011.02329/) (2020)
- [The Intel Neuromorphic DNS Challenge](https://mlsanity.com/paper/2303.09503/) (2023)

## Built on by (most cited first, 1 of 1)

- [Exploring the Benefits of Tokenization of Discrete Acoustic Units](https://mlsanity.com/paper/2406.05547/) (2024)

## Abstract (arXiv)

Neurons in the brain communicate information via punctual events called spikes. The timing of spikes is thought to carry rich information, but it is not clear how to leverage this in digital systems. We demonstrate that event-based encoding is efficient for audio compression. To build this event-based representation we use a deep binary auto-encoder, and under high sparsity pressure, the model enters a regime where the binary event matrix is stored more efficiently with sparse matrix storage algorithms. We test this on the large MAESTRO dataset of piano recordings against vector quantized auto-encoders. Not only does our "Spiking Music compression" algorithm achieve a competitive compression/reconstruction trade-off, but selectivity and synchrony between encoded events and piano key strikes emerge without supervision in the sparse regime.