mlsanityData to 7 Oct 2026

RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction

arXiv 2608.09331 ↗First posted 10 Aug 2026cs.SD cs.AIcited by 0

On the map: Retrieval-augmented generation → Conversational and speech RAG

Abstract (arXiv)

Brain-to-audio reconstruction is limited by prior domination: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator's sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from 0.14--0.18 for direct generation, near the 0.10 chance level, to 0.40--0.43, comparable to retrieval. It also reduces Fr'echet Audio Distance by roughly an order of magnitude, from 13.49 to 1.25 for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 51 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

35 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Retrieval-Augmented Text-to-Audio Generation14 Sep

Other references · 34

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022Flow Matching for Generative Modelingcited by 5,426
  10. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
22 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  3. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  4. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  5. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  7. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  8. 2023Simple and Controllable Music Generationcited by 480
  9. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 214
  10. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  11. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
  12. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
  13. 2022Decoding speech perception from non-invasive brain recordingscited by 129
  14. 2024MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datacited by 105
  15. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  16. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  17. 2022Semi-Parametric Neural Image Synthesiscited by 36
  18. 2024NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulationcited by 33
  19. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  20. 2023Sound reconstruction from human brain activity via a generative model with brain-like auditory featurescited by 5
  21. 2024Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRIcited by 2
  22. 2024R&B -- Rhythm and Brain: Cross-subject Decoding of Music from Human Brain Activitycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Relational Synthesis: Structure-Mediated Concatenative Synthesis for Foley and Retrieval-Augmented Audio Generation
  2. 2025Perceptually Aligning Representations of Music via Noise-Augmented Autoencoders
  3. 2025Learning to Upsample and Upmix Audio in the Latent Domain
  4. 2026On the Geometry of Music Bandwidth Extension in Latent Spaces of Audio Codecs
  5. 2025Music2Latent2: Audio Compression with Summary Embeddings and Autoregressive Decoding
  6. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond
  7. 2026Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation ↗
  8. 2026EvoAudio: Recursive Self-Improvement for Audio Understanding