# Neuroscience-Inspired Analyses of Visual Interestingness in Multimodal Transformers

arXiv: https://arxiv.org/abs/2605.08188 · HTML: https://mlsanity.com/paper/2605.08188/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 64 references that are in the map.
- First posted on arXiv 5 May 2026.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Understanding Reasoning in Thinking Language Models via Steering Vectors](https://mlsanity.com/paper/2506.18167/) (2025)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [DeepGaze II: Reading fixations from deep features trained on object recognition](https://mlsanity.com/paper/1610.01563/) (2016)
- [How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks](https://mlsanity.com/paper/1811.01753/) (2018)
- [Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment](https://mlsanity.com/paper/2206.08614/) (2022)
- [Refusal Behavior in Large Language Models: A Nonlinear Perspective](https://mlsanity.com/paper/2501.08145/) (2025)
- [A statistical method for analyzing and comparing spatiotemporal cortical activation patterns](https://mlsanity.com/paper/1611.07677/) (2016)
- [Linear Personality Probing and Steering in LLMs: A Big Five Study](https://mlsanity.com/paper/2512.17639/) (2025)
- ["Understanding AI": Semantic Grounding in Large Language Models](https://mlsanity.com/paper/2402.10992/) (2024)
- [Commonly Interesting Images](https://mlsanity.com/paper/2409.16736/) (2024)
- [Cognition spaces: natural, artificial, and hybrid](https://mlsanity.com/paper/2601.12837/) (2026)
- [Different Spectral Representations in Optimized Artificial Neural Networks and Brains](https://mlsanity.com/paper/2208.10576/) (2022)
- [Visual Interestingness Decoded: How GPT-4o Mirrors Human Interests](https://mlsanity.com/paper/2510.13316/) (2025)

## Abstract (arXiv)

Human attention is the gateway to conscious perception, memory and decision-making. However, its role in modern transformer models remains largely unexplored. As these systems increasingly influence what people see, prefer and buy, the question arises as to whether they encode principles of human interest or merely exploit large-scale correlations. Addressing this issue is crucial for understanding cognition and ensuring the responsible use of AI in communication and marketing. In order to address this issue, the concept of visual interest was examined within the multimodal vision-language-model Qwen3-VL-8B, using a pre-defined Common Interestingness (CI) score derived from large-scale human engagement data on the photo-sharing platform Flickr. Here, we analyzed internal representations across vision and language components using methods from the neurosciences. Our analyses revealed that CI information is linearly decodable from final-layer embeddings, indicating that it is aligned with human-derived measures of visual interestingness. Dimensionality reduction and Generalized Discrimination Value (GDV) analyses demonstrate that CI-related hidden representations emerge in intermediate vision transformer layers and becomes progressively more distinguishable across language model layers. Concept vectors derived using geometric, probe, and Sparse Auto-Encoder based methods converge in higher layers, as confirmed by representational similarity analysis. This indicates a robust and structured encoding of visual interestingness without explicit supervision. Future work will seek to identify shared computational principles linking human brain dynamics and transformer architectures, with the ultimate goal of uncovering the organizing mechanisms that give rise to attention and interest in both biological and artificial systems.