mlsanityData to 14 Oct 2026

Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion

arXiv 2411.09723 ↗First posted 14 Nov 2024cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities are demonstrated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities. The results highlight the model's ability to accurately capture semantic information across different brain imaging techniques, illustrating its potential in decoding, encoding, and modality conversion tasks.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

17 of 39 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 160
  3. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 144
  4. 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
  5. 2016Deep Learning Human Mind for Automated Visual Classificationcited by 77
  6. 2023Scaling laws for language encoding models in fMRIcited by 68
  7. 2023Brain decoding: toward real-time reconstruction of visual perceptioncited by 66
  8. 2023DreamDiffusion: Generating High-Quality Images from Brain EEG Signalscited by 56
  9. 2023DREAM: Visual Decoding from Reversing Human Visual Systemcited by 44
  10. 2023The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenescited by 36
  11. 2023Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)cited by 21
  12. 2022Semantic Brain Decoding: from fMRI to conceptually similar image reconstruction of visual stimulicited by 12
5 more
  1. 2023The Algonauts Project 2023 Challenge: UARK-UAlbany Team Solutioncited by 11
  2. 2023Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion modelscited by 10
  3. 2023Memory Encoding Modelcited by 7
  4. 2020Correct block-design experiments mitigate temporal correlation bias in EEG classificationcited by 4
  5. 2018Training on the test set? An analysis of Spampinato et al. [31]cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Transformer brain encoders explain human high-level visual responsescited by 10
  2. 2026Brain4FMs: A Benchmark of Foundation Models for Electrical Brain Signalcited by 7
  3. 2025Foundation Models for Cross-Domain EEG Analysis Application: A Surveycited by 4
  4. 2025Shifting Attention to You: Personalized Brain-Inspired AI Modelscited by 1
  5. 2025Bridging Brain with Foundation Models through Self-Supervised Learningcited by 1
  6. 2025On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustnesscited by 1
  7. 2025Towards Interpretable Visual Decoding with Attention to Brain Representationscited by 1
  8. 2026EEG2Vision: A Multimodal EEG-Based Framework for 2D Visual Reconstruction in Cognitive Neurosciencecited by 1
  9. 2026A Roadmap for MEG Foundation Modelscited by 0