# Naturalistic Music Decoding from EEG Data via Latent Diffusion Models

arXiv: https://arxiv.org/abs/2405.09062 · HTML: https://mlsanity.com/paper/2405.09062/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 15 May 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Adapting Frechet Audio Distance for Generative Music Evaluation](https://mlsanity.com/paper/2311.01616/) (2023)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [Mousai: Text-to-Music Generation with Long-Context Latent Diffusion](https://mlsanity.com/paper/2301.11757/) (2023)
- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [DreamDiffusion: Generating High-Quality Images from Brain EEG Signals](https://mlsanity.com/paper/2306.16934/) (2023)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [Full-band General Audio Synthesis with Score-based Diffusion](https://mlsanity.com/paper/2210.14661/) (2022)
- [StemGen: A music generation model that listens](https://mlsanity.com/paper/2312.08723/) (2023)
- [Controllable Mind Visual Diffusion Model](https://mlsanity.com/paper/2305.10135/) (2023)
- [SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis](https://mlsanity.com/paper/2310.15247/) (2023)
- [InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models](https://mlsanity.com/paper/2308.14360/) (2023)
- [Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution](https://mlsanity.com/paper/2210.15793/) (2022)

## Built on by (most cited first, 8 of 8)

- [Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models](https://mlsanity.com/paper/2410.02780/) (2024)
- [Foundation Models for Cross-Domain EEG Analysis Application: A Survey](https://mlsanity.com/paper/2508.15716/) (2025)
- [Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordings](https://mlsanity.com/paper/2412.15560/) (2024)
- [A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond](https://mlsanity.com/paper/2502.12048/) (2025)
- [Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions](https://mlsanity.com/paper/2603.12296/) (2026)
- [MindMelody: A Closed-Loop EEG-Driven System for Personalized Music Intervention](https://mlsanity.com/paper/2605.01235/) (2026)
- [Channel-Preserving Representation Alignment for EEG-to-Music Reconstruction](https://mlsanity.com/paper/2606.04040/) (2026)
- [RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction](https://mlsanity.com/paper/2608.09331/) (2026)

## Abstract (arXiv)

In this article, we explore the potential of using latent diffusion models, a family of powerful generative models, for the task of reconstructing naturalistic music from electroencephalogram (EEG) recordings. Unlike simpler music with limited timbres, such as MIDI-generated tunes or monophonic pieces, the focus here is on intricate music featuring a diverse array of instruments, voices, and effects, rich in harmonics and timbre. This study represents an initial foray into achieving general music reconstruction of high-quality using non-invasive EEG data, employing an end-to-end training approach directly on raw data without the need for manual pre-processing and channel selection. We train our models on the public NMED-T dataset and perform quantitative evaluation proposing neural embedding-based metrics. Our work contributes to the ongoing research in neural decoding and brain-computer interfaces, offering insights into the feasibility of using EEG data for complex auditory information reconstruction.