# Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

arXiv: https://arxiv.org/abs/2403.07721 · HTML: https://mlsanity.com/paper/2403.07721/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Decoding Natural Images from EEG for Object Recognition](https://mlsanity.com/paper/2308.13234/) (2023)
- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features](https://mlsanity.com/paper/2210.06756/) (2022)
- [Transfer Learning for EEG-Based Brain-Computer Interfaces: A Review of Progress Made Since 2016](https://mlsanity.com/paper/2004.06286/) (2020)
- [Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features](https://mlsanity.com/paper/1810.10974/) (2018)
- [BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding](https://mlsanity.com/paper/2302.12971/) (2023)
- [Transfer Learning for Motor Imagery Based Brain-Computer Interfaces: A Complete Pipeline](https://mlsanity.com/paper/2007.03746/) (2020)
- [Object classification from randomized EEG trials](https://mlsanity.com/paper/2004.06046/) (2020)

## Built on by (most cited first, 30 of 81)

- [Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior](https://mlsanity.com/paper/2503.04207/) (2025)
- [CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Information](https://mlsanity.com/paper/2412.10489/) (2024)
- [Neuro-3D: Towards 3D Visual Decoding from EEG Signals](https://mlsanity.com/paper/2411.12248/) (2024)
- [Visual Neural Decoding via Improved Visual-EEG Semantic Consistency](https://mlsanity.com/paper/2408.06788/) (2024)
- [AdaBrain-Bench: Benchmarking Brain Foundation Models for Brain-Computer Interface Applications](https://mlsanity.com/paper/2507.09882/) (2025)
- [Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding](https://mlsanity.com/paper/2412.17337/) (2024)
- [NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment](https://mlsanity.com/paper/2511.06836/) (2025)
- [VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models](https://mlsanity.com/paper/2412.00156/) (2024)
- [Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models](https://mlsanity.com/paper/2410.02780/) (2024)
- [CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing](https://mlsanity.com/paper/2503.06940/) (2025)
- [NECOMIMI: Neural-Cognitive Multimodal EEG-informed Image Generation with Diffusion Models](https://mlsanity.com/paper/2410.00712/) (2024)
- [Human-Aligned Image Models Improve Visual Decoding from the Brain](https://mlsanity.com/paper/2502.03081/) (2025)
- [Dual-TSST: A Dual-Branch Temporal-Spectral-Spatial Transformer Model for EEG Decoding](https://mlsanity.com/paper/2409.03251/) (2024)
- [WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities](https://mlsanity.com/paper/2510.00032/) (2025)
- [ViEEG: Hierarchical Visual Neural Representation for EEG Brain Decoding](https://mlsanity.com/paper/2505.12408/) (2025)
- [Aligning What EEG Can See: Structural Representations for Brain-Vision Matching](https://mlsanity.com/paper/2603.07077/) (2026)
- [HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment](https://mlsanity.com/paper/2603.22721/) (2026)
- [Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex](https://mlsanity.com/paper/2505.15813/) (2025)
- [EEG2TEXT-CN: An Exploratory Study of Open-Vocabulary Chinese Text-EEG Alignment via Large Language Model and Contrastive Learning on ChineseEEG](https://mlsanity.com/paper/2506.00854/) (2025)
- [Foundation Models for Cross-Domain EEG Analysis Application: A Survey](https://mlsanity.com/paper/2508.15716/) (2025)
- [Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding](https://mlsanity.com/paper/2601.21948/) (2026)
- [Subject-Aware Multi-Granularity Alignment for Zero-Shot EEG-to-Image Retrieval](https://mlsanity.com/paper/2604.17782/) (2026)
- [A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond](https://mlsanity.com/paper/2502.12048/) (2025)
- [Category-aware EEG image generation based on wavelet transform and contrast semantic loss](https://mlsanity.com/paper/2505.24301/) (2025)
- [Transformer-based EEG Decoding: A Survey](https://mlsanity.com/paper/2507.02320/) (2025)
- [Spatial-Functional awareness Transformer-based graph archetype contrastive learning for Decoding Visual Neural Representations from EEG](https://mlsanity.com/paper/2509.24761/) (2025)
- [Learning Brain Representation with Hierarchical Visual Embeddings](https://mlsanity.com/paper/2602.07495/) (2026)
- [Autoregressive Visual Decoding from EEG Signals](https://mlsanity.com/paper/2602.22555/) (2026)
- [SATTC: Structure-Aware Label-Free Test-Time Calibration for Cross-Subject EEG-to-Image Retrieval](https://mlsanity.com/paper/2603.20738/) (2026)
- [Integrating Biological and Machine Intelligence: Attention Mechanisms in Brain-Computer Interfaces](https://mlsanity.com/paper/2502.19281/) (2025)

## Abstract (arXiv)

How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of visual decoding and reconstruction based on functional Magnetic Resonance Imaging (fMRI). However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for visual decoding based on electroencephalography (EEG). In this study, we present an end-to-end EEG-based visual reconstruction zero-shot framework, consisting of a tailored brain encoder, called the Adaptive Thinking Mapper (ATM), which projects neural signals from different sources into the shared subspace as the clip embedding, and a two-stage multi-pipe EEG-to-image generation strategy. In stage one, EEG is embedded to align the high-level clip embedding, and then the prior diffusion model refines EEG embedding into image priors. A blurry image also decoded from EEG for maintaining the low-level feature. In stage two, we input both the high-level clip embedding, the blurry image and caption from EEG latent to a pre-trained diffusion model. Furthermore, we analyzed the impacts of different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. The experimental results indicate that our EEG-based visual zero-shot framework achieves SOTA performance in classification, retrieval and reconstruction, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. Our code is available at https://github.com/ncclab-sustech/EEG_Image_decode.