mlsanityData to 7 Oct 2026

Brain2Music: Reconstructing Music from Human Brain Activity

arXiv 2307.11078 ↗First posted 20 Jul 2023q-bio.NC cs.LG cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

The process of reconstructing experiences from human brain activity offers a unique lens into how the brain interprets and represents the world. In this paper, we introduce a method for reconstructing music from brain activity, captured using functional magnetic resonance imaging (fMRI). Our approach uses either music retrieval or the MusicLM music generation model conditioned on embeddings derived from fMRI data. The generated music resembles the musical stimuli that human subjects experienced, with respect to semantic properties like genre, instrumentation, and mood. We investigate the relationship between different components of MusicLM and brain activity through a voxel-wise encoding modeling analysis. Furthermore, we discuss which brain regions represent information derived from purely textual descriptions of music stimuli. We provide supplementary material including examples of the reconstructed music at https://google-research.github.io/seanet/brain2music

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 71 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

25 of 71 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  6. 2022High Fidelity Neural Audio Compressioncited by 673
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  9. 2020Jukebox: A Generative Model for Musiccited by 535
  10. 2023Simple and Controllable Music Generationcited by 480
  11. 2023MusicLM: Generating Music From Textcited by 442
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
13 more
  1. 2016FMA: A Dataset For Music Analysiscited by 205
  2. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  3. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  4. 2014Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathwaycited by 93
  5. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  6. 2022Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolutioncited by 78
  7. 2021LEAF: A Learnable Frontend for Audio Classificationcited by 69
  8. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  9. 2023SingSong: Generating musical accompaniments from singingcited by 54
  10. 2023Efficient Neural Music Generationcited by 50
  11. 2023Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputscited by 23
  12. 2023Sound reconstruction from human brain activity via a generative model with brain-like auditory featurescited by 5
  13. 2016Brains on Beatscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 2; 2026: 1.

10 papers.

  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  2. 2023Brain-Conditional Multimodal Synthesis: A Survey and Taxonomycited by 15
  3. 2024A Survey on Multimodal Wearable Sensor-based Human Action Recognitioncited by 10
  4. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  5. 2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
  6. 2024Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordingscited by 3
  7. 2024Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordingscited by 3
  8. 2026Linguistics and Human Brain: A Perspective of Computational Neurosciencecited by 3
  9. 2024Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRIcited by 2
  10. 2025Whole-brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decodingcited by 2