# Neurons: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction

arXiv: https://arxiv.org/abs/2503.11167 · HTML: https://mlsanity.com/paper/2503.11167/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data](https://mlsanity.com/paper/2403.11207/) (2024)
- [Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity](https://mlsanity.com/paper/2305.11675/) (2023)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [CVPR 2023 Text Guided Video Editing Competition](https://mlsanity.com/paper/2310.16003/) (2023)
- [UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity](https://mlsanity.com/paper/2308.07428/) (2023)
- [MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction](https://mlsanity.com/paper/2404.12630/) (2024)
- [NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction](https://mlsanity.com/paper/2410.19452/) (2024)
- [A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity](https://mlsanity.com/paper/2206.03544/) (2022)
- [Brain Decodes Deep Nets](https://mlsanity.com/paper/2312.01280/) (2023)
- [Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity](https://mlsanity.com/paper/2405.03280/) (2024)
- [Lite-Mind: Towards Efficient and Robust Brain Representation Network](https://mlsanity.com/paper/2312.03781/) (2023)

## Built on by (most cited first, 6 of 6)

- [ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding](https://mlsanity.com/paper/2510.27128/) (2025)
- [A Cognitive Process-Inspired Architecture for Subject-Agnostic Brain Visual Decoding](https://mlsanity.com/paper/2511.02565/) (2025)
- [SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance](https://mlsanity.com/paper/2602.21819/) (2026)
- [Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding](https://mlsanity.com/paper/2603.03312/) (2026)
- [Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction](https://mlsanity.com/paper/2605.14569/) (2026)
- [Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability](https://mlsanity.com/paper/2610.13763/) (2026)

## Abstract (arXiv)

Decoding visual stimuli from neural activity is essential for understanding the human brain. While fMRI methods have successfully reconstructed static images, fMRI-to-video reconstruction faces challenges due to the need for capturing spatiotemporal dynamics like motion and scene transitions. Recent approaches have improved semantic and perceptual alignment but struggle to integrate coarse fMRI data with detailed visual features. Inspired by the hierarchical organization of the visual system, we propose NEURONS, a novel framework that decouples learning into four correlated sub-tasks: key object segmentation, concept recognition, scene description, and blurry video reconstruction. This approach simulates the visual cortex's functional specialization, allowing the model to capture diverse video content. In the inference stage, NEURONS generates robust conditioning signals for a pre-trained text-to-video diffusion model to reconstruct the videos. Extensive experiments demonstrate that NEURONS outperforms state-of-the-art baselines, achieving solid improvements in video consistency (26.6%) and semantic-level accuracy (19.1%). Notably, NEURONS shows a strong functional correlation with the visual cortex, highlighting its potential for brain-computer interfaces and clinical applications. Code and model weights are available at: https://github.com/xmed-lab/NEURONS.