Perceptual misalignment of texture representations in convolutional neural networks
Not on the map. Why?
Abstract (arXiv)
Mathematical modeling of visual textures traces back to Julesz's intuition that texture perception in humans is based on local correlations between image features. An influential approach for texture analysis and generation generalizes this notion to linear correlations between the nonlinear features computed by convolutional neural networks (CNNs), compiled into Gram matrices. Given that CNNs are often used as models for the visual system, it is natural to ask whether such "texture representations" spontaneously align with the textures' perceptual content, and in particular whether those CNNs that are regarded as better models for the visual system also possess more human-like texture representations. Here we quantify the perceptual content captured by feature correlations computed for a diverse pool of CNNs, and we compare it to the models' perceptual alignment with the mammalian visual system as measured by Brain-Score. Surprisingly, we find that there is no connection between conventional measures of CNN quality as a model of the visual system and its alignment with human texture perception. We conclude that texture perception involves mechanisms that are distinct from those that are commonly modeled using approaches based on CNNs trained on object recognition, possibly depending on the integration of contextual information.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 83 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
24 of 83 references.
References · 24
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
- 2013Describing Textures in the Wildcited by 2,110
- 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
- 2017Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesiscited by 215
- 2021Partial success in closing the gap between human and machine visioncited by 185
12 more
- 2014Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathwaycited by 93
- 2017Diversified Texture Synthesis with Feed-forward Networkscited by 59
- 2021Shape or Texture: Understanding Discriminative Features in CNNscited by 52
- 2018From BoW to CNN: Two Decades of Texture Representation for Texture Classificationcited by 39
- 2020Texture Interpolation for Probing Visual Perceptioncited by 15
- 2019Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texturecited by 14
- 2023Are Deep Neural Networks Adequate Behavioural Models of Human Visual Perception?cited by 10
- 2016Texture Synthesis Through Convolutional Neural Networks and Spectrum Constraintscited by 8
- 2016Incorporating long-range consistency in CNN-based texture generationcited by 8
- 2024Infinite Texture: Text-guided High Resolution Diffusion Texture Synthesiscited by 5
- 2022Paying U-Attention to Textures: Multi-Stage Hourglass Vision Transformer for Universal Texture Synthesiscited by 4
- 2022Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networkscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.