Modality-Agnostic fMRI Decoding of Vision and Language
Not on the map. Why?
Abstract (arXiv)
Previous studies have shown that it is possible to map brain activation data of subjects viewing images onto the feature representation space of not only vision models (modality-specific decoding) but also language models (cross-modal decoding). In this work, we introduce and use a new large-scale fMRI dataset (~8,500 trials per subject) of people watching both images and text descriptions of such images. This novel dataset enables the development of modality-agnostic decoders: a single decoder that can predict which stimulus a subject is seeing, irrespective of the modality (image or text) in which the stimulus is presented. We train and evaluate such decoders to map brain signals onto stimulus representations from a large range of publicly available vision, language and multimodal (vision+language) models. Our findings reveal that (1) modality-agnostic decoders perform as well as (and sometimes even better than) modality-specific decoders (2) modality-agnostic decoders mapping brain data onto representations from unimodal models perform as well as decoders relying on multimodal representations (3) while language and low-level visual (occipital) brain regions are best at decoding text and image stimuli, respectively, high-level visual (temporal) regions perform well on both stimulus types.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 53 references that are in the map. First posted on arXiv 18 Mar 2024.
Builds on
24 of 53 references.
References · 24
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
- 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
12 more
- 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
- 2023Natural scene reconstruction from fMRI signals using generative latent diffusioncited by 97
- 2023Brain decoding: toward real-time reconstruction of visual perceptioncited by 64
- 2018BOLD5000: A public fMRI dataset of 5000 imagescited by 63
- 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
- 2023DREAM: Visual Decoding from Reversing Human Visual Systemcited by 44
- 2018Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networkscited by 43
- 2023Brain encoding models based on multimodal transformers can transfer across language and visioncited by 31
- 2018Interpreting Encoding and Decoding Modelscited by 21
- 2023Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Braincited by 9
- 2023Mapping Brains with Language Models: A Surveycited by 8
- 2018Describing Semantic Representations of Brain Activity Evoked by Visual Stimulicited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.