# The Roles of Contextual Semantic Relevance Metrics in Human Visual Processing

arXiv: https://arxiv.org/abs/2410.09921 · HTML: https://mlsanity.com/paper/2410.09921/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 73 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Smoothing parameter and model selection for general smooth models](https://mlsanity.com/paper/1511.03864/) (2015)
- [Not just a matter of semantics: the relationship between visual similarity and semantic similarity](https://mlsanity.com/paper/1811.07120/) (2018)
- [Attention-aware semantic relevance predicting Chinese sentence reading](https://mlsanity.com/paper/2403.18542/) (2024)
- [Visual Similarity Attention](https://mlsanity.com/paper/1911.07381/) (2019)
- [Differential contributions of machine learning and statistical analysis to language and cognitive sciences](https://mlsanity.com/paper/2404.14052/) (2024)

## Abstract (arXiv)

Semantic relevance metrics can capture both the inherent semantics of individual objects and their relationships to other elements within a visual scene. Numerous previous research has demonstrated that these metrics can influence human visual processing. However, these studies often did not fully account for contextual information or employ the recent deep learning models for more accurate computation. This study investigates human visual perception and processing by introducing the metrics of contextual semantic relevance. We evaluate semantic relationships between target objects and their surroundings from both vision-based and language-based perspectives. Testing a large eye-movement dataset from visual comprehension, we employ state-of-the-art deep learning techniques to compute these metrics and analyze their impacts on fixation measures on human visual processing through advanced statistical models. These metrics could also simulate top-down and bottom-up processing in visual perception. This study further integrates vision-based and language-based metrics into a novel combined metric, addressing a critical gap in previous research that often treated visual and semantic similarities separately. Results indicate that all metrics could precisely predict fixation measures in visual perception and processing, but with distinct roles in prediction. The combined metric outperforms other metrics, supporting theories that emphasize the interaction between semantic and visual information in shaping visual perception/processing. This finding aligns with growing recognition of the importance of multi-modal information processing in human cognition. These insights enhance our understanding of cognitive mechanisms underlying visual processing and have implications for developing more accurate computational models in fields such as cognitive science and human-computer interaction.