mlsanityData to 7 Oct 2026

Multimodal neural networks better explain multivoxel patterns in the hippocampus

arXiv 2201.11517 ↗First posted 11 Dec 2021q-bio.NC cs.LG cs.NEcited by 3

Not on the map. Why?

Abstract (arXiv)

The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et at., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 11 Dec 2021.

Builds on

14 of 30 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  7. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  8. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  9. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  10. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  11. 2020Learning Visual Representations with Caption Annotationscited by 123
  12. 2015Generic decoding of seen and imagined objects using hierarchical visual featurescited by 119
2 more
  1. 2014Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathwaycited by 93
  2. 2021Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamicscited by 19

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1.

3 papers.

  1. 2022Mind Reader: Reconstructing complex images from brain activitiescited by 85
  2. 2023Brain Captioning: Decoding human brain activity into images and textcited by 33
  3. 2024Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learningcited by 4