# Off-the-Shelf Language-Vision Encoders Predict Human Gaze Preferences in Visual World Experiments

arXiv: https://arxiv.org/abs/2608.07282 · HTML: https://mlsanity.com/paper/2608.07282/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [Approximate Attributions for Off-the-Shelf Siamese Transformers](https://mlsanity.com/paper/2402.02883/) (2024)

## Abstract (arXiv)

A central question for computational psycholinguistics is to what extent neural language models can serve as cognitively adequate models of human language processing. However, work has been overwhelmingly focused on the unimodal case of written or spoken language. In contrast, multimodal paradigms, like visual world experiments that present participants with both visual and linguistic input simultaneously, have been neglected. In this paper, we present a novel approach that predicts patterns of gaze preferences in visual world studies. To do so, it derives target preferences by applying an attribution method to a simple bi-modal CLIP bi-encoder. We demonstrate the ability of this approach to qualitatively replicate the gaze patterns found in a seminal English visual world study demonstrating human predictive processing. It does so without a generative architecture and without the need for fine-tuning, despite not being trained for the task.