# Brain decoding: toward real-time reconstruction of visual perception

arXiv: https://arxiv.org/abs/2310.19812 · HTML: https://mlsanity.com/paper/2310.19812/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 55 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Deep learning-based electroencephalography analysis: a systematic review](https://mlsanity.com/paper/1901.05498/) (2019)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [MOABB: Trustworthy algorithm benchmarking for BCIs](https://mlsanity.com/paper/1805.06427/) (2018)
- [Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features](https://mlsanity.com/paper/1810.10974/) (2018)
- [Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks](https://mlsanity.com/paper/1810.03856/) (2018)
- [UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity](https://mlsanity.com/paper/2308.07428/) (2023)
- [Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data](https://mlsanity.com/paper/2206.11417/) (2022)
- [Controllable Mind Visual Diffusion Model](https://mlsanity.com/paper/2305.10135/) (2023)
- [Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimuli](https://mlsanity.com/paper/2212.06726/) (2022)
- [Measuring and modeling the motor system with machine learning](https://mlsanity.com/paper/2103.11775/) (2021)

## Built on by (most cited first, 30 of 66)

- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion](https://mlsanity.com/paper/2403.07721/) (2024)
- [Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior](https://mlsanity.com/paper/2503.04207/) (2025)
- [CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Information](https://mlsanity.com/paper/2412.10489/) (2024)
- [BrainOmni: A Brain Foundation Model for Unified EEG and MEG Signals](https://mlsanity.com/paper/2505.18185/) (2025)
- [Scaling laws for decoding images from brain activity](https://mlsanity.com/paper/2501.15322/) (2025)
- [Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)](https://mlsanity.com/paper/2410.07507/) (2024)
- [NeuralBench: A Unifying Framework to Benchmark NeuroAI Models](https://mlsanity.com/paper/2605.08495/) (2026)
- [Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects](https://mlsanity.com/paper/2311.02332/) (2023)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment](https://mlsanity.com/paper/2511.06836/) (2025)
- [Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream](https://mlsanity.com/paper/2411.05712/) (2024)
- [A foundation model of vision, audition, and language for in-silico neuroscience](https://mlsanity.com/paper/2605.04326/) (2026)
- [Aligning brain functions boosts the decoding of visual semantics in novel subjects](https://mlsanity.com/paper/2312.06467/) (2023)
- [Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion](https://mlsanity.com/paper/2411.09723/) (2024)
- [SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning](https://mlsanity.com/paper/2508.10298/) (2025)
- [Toward Generalizing Visual Brain Decoding to Unseen Subjects](https://mlsanity.com/paper/2410.14445/) (2024)
- [Human-Aligned Image Models Improve Visual Decoding from the Brain](https://mlsanity.com/paper/2502.03081/) (2025)
- [Geometric Neural Network based on Phase Space for BCI-EEG decoding](https://mlsanity.com/paper/2403.05645/) (2024)
- [Adaptive Intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems](https://mlsanity.com/paper/2411.15234/) (2024)
- [Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRI](https://mlsanity.com/paper/2505.14556/) (2025)
- [Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation](https://mlsanity.com/paper/2505.17099/) (2025)
- [Through-Wall Imaging based on WiFi Channel State Information](https://mlsanity.com/paper/2401.17417/) (2024)
- [Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex](https://mlsanity.com/paper/2505.15813/) (2025)
- [Neural-Driven Image Editing](https://mlsanity.com/paper/2507.05397/) (2025)
- [MEGConformer: Conformer-Based MEG Decoder for Robust Speech and Phoneme Classification](https://mlsanity.com/paper/2512.01443/) (2025)
- [Subject-Aware Multi-Granularity Alignment for Zero-Shot EEG-to-Image Retrieval](https://mlsanity.com/paper/2604.17782/) (2026)
- [OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens](https://mlsanity.com/paper/2604.18827/) (2026)
- [Emergence and Function of Abstract Representations in Self-Supervised Transformers](https://mlsanity.com/paper/2312.05361/) (2023)
- [CrossPT-EEG: A Benchmark for Cross-Participant and Cross-Time Generalization of EEG-based Visual Decoding](https://mlsanity.com/paper/2406.07151/) (2024)

## Abstract (arXiv)

In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution (≈0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution (≈5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that high-level visual features can be decoded from MEG signals, although the same approach applied to 7T fMRI also recovers better low-level features. Overall, these results, while preliminary, provide an important step towards the decoding -- in real-time -- of the visual processes continuously unfolding within the human brain.