Off-the-Shelf Language-Vision Encoders Predict Human Gaze Preferences in Visual World Experiments

A central question for computational psycholinguistics is to what extent neural language models can serve as cognitively adequate models of human language processing. However, work has been overwhelmingly focused on the unimodal case of written or spoken language. In contrast, multimodal paradigms, like visual world experiments that present participants with both visual and linguistic input simultaneously, have been neglected. In this paper, we present a novel approach that predicts patterns of gaze preferences in visual world studies. To do so, it derives target preferences by applying an attribution method to a simple bi-modal CLIP bi-encoder. We demonstrate the ability of this approach to qualitatively replicate the gaze patterns found in a seminal English visual world study demonstrating human predictive processing. It does so without a generative architecture and without the need for fine-tuning, despite not being trained for the task.