# Language learning shapes visual category-selectivity in deep neural networks

arXiv: https://arxiv.org/abs/2502.16456 · HTML: https://mlsanity.com/paper/2502.16456/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 54 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs](https://mlsanity.com/paper/1909.06161/) (2019)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition](https://mlsanity.com/paper/1406.3284/) (2014)
- [Modeling Category-Selective Cortical Regions with Topographic Variational Autoencoders](https://mlsanity.com/paper/2110.13911/) (2021)
- [End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions](https://mlsanity.com/paper/2308.09431/) (2023)
- [Achieving More Human Brain-Like Vision via Human EEG Representational Alignment](https://mlsanity.com/paper/2401.17231/) (2024)
- [Generate your neural signals from mine: individual-to-individual EEG converters](https://mlsanity.com/paper/2304.10736/) (2023)
- [Teaching CORnet Human fMRI Representations for Enhanced Model-Brain Alignment](https://mlsanity.com/paper/2407.10414/) (2024)
- [Cracking the neural code for word recognition in convolutional neural networks](https://mlsanity.com/paper/2403.06159/) (2024)

## Abstract (arXiv)

Category-selective regions in the human brain-such as the fusiform face area (FFA), extrastriate body area (EBA), parahippocampal place area (PPA), and visual word form area (VWFA)-support high-level visual recognition. Here, we investigate whether artificial neural networks (ANNs) exhibit analogous category-selective neurons and how these representations are shaped by language experience. Using an fMRI-inspired functional localizer approach, we identified face-, body-, place-, and word-selective neurons in deep networks presented with category images and scrambled controls. Both the purely visual ResNet and a linguistically supervised Lang-Learned ResNet contained category-selective neurons that increased in proportion across layers. However, compared to the vision-only model, the Lang-Learned ResNet showed a greater number but lower specificity of category-selective neurons, along with reduced spatial localization and attenuated activation strength-indicating a shift toward more distributed, semantically aligned coding. These effects were replicated in the large-scale vision-language model CLIP. Together, our findings reveal that language experience systematically reorganizes visual category representations in ANNs, providing a computational parallel to how linguistic context may shape categorical organization in the human brain.