# Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

arXiv: https://arxiv.org/abs/2404.19438 · HTML: https://mlsanity.com/paper/2404.19438/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2024.

## Built on by (most cited first, 16 of 16)

- [Scaling laws for decoding images from brain activity](https://mlsanity.com/paper/2501.15322/) (2025)
- [MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI](https://mlsanity.com/paper/2503.02351/) (2025)
- [Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer](https://mlsanity.com/paper/2510.25976/) (2025)
- [HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment](https://mlsanity.com/paper/2603.22721/) (2026)
- [Exploring The Visual Feature Space for Multimodal Neural Decoding](https://mlsanity.com/paper/2505.15755/) (2025)
- [Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing](https://mlsanity.com/paper/2503.10663/) (2025)
- [Multigranular Evaluation for Brain Visual Decoding](https://mlsanity.com/paper/2507.07993/) (2025)
- [From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structure](https://mlsanity.com/paper/2507.16389/) (2025)
- [SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding](https://mlsanity.com/paper/2503.06437/) (2025)
- [Probability Distribution Alignment and Low-Rank Weight Decomposition for Source-Free Domain Adaptive Brain Decoding](https://mlsanity.com/paper/2504.09109/) (2025)
- [MindPilot: Closed-loop Visual Stimulation Optimization for Brain Modulation with EEG-guided Diffusion](https://mlsanity.com/paper/2602.10552/) (2026)
- [NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity](https://mlsanity.com/paper/2604.09817/) (2026)
- [StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation](https://mlsanity.com/paper/2605.02586/) (2026)
- [MindAdapter: Few-Shot Parameter-Efficient Residual Calibration of Cross-Subject Brain-to-Visual Decoding Models](https://mlsanity.com/paper/2605.24679/) (2026)
- [NeurIPS: Neuro-anatomical Inductive Priors for Sphere-based Brain Decoding](https://mlsanity.com/paper/2605.24993/) (2026)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)

## Abstract (arXiv)

Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition but faces challenges due to individual differences and complex neural signal representations. Traditional methods often require customized models and extensive trials, lacking interpretability in visual reconstruction tasks. Our framework integrates 3D brain structures with visual semantics using a Vision Transformer 3D. This unified feature extractor efficiently aligns fMRI features with multiple levels of visual embeddings, eliminating the need for subject-specific models and allowing extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with Large Language Models (LLMs). Additionally, we have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development. Integrating with LLMs enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. Our approach demonstrates superior performance across these tasks, precisely identifying language-based concepts within brain signals, enhancing interpretability, and providing deeper insights into neural processes. These advances significantly broaden the applicability of non-invasive brain decoding in neuroscience and human-computer interaction, setting the stage for advanced brain-computer interfaces and cognitive models.