# GazBy: Gaze-Based BERT Model to Incorporate Human Attention in Neural Information Retrieval

arXiv: https://arxiv.org/abs/2207.01674 · HTML: https://mlsanity.com/paper/2207.01674/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2022.

## Built on by (most cited first, 2 of 2)

- [WebQAmGaze: A Multilingual Webcam Eye-Tracking-While-Reading Dataset](https://mlsanity.com/paper/2303.17876/) (2023)
- [Describing Images textitFast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes](https://mlsanity.com/paper/2402.01352/) (2024)

## Abstract (arXiv)

This paper is interested in investigating whether human gaze signals can be leveraged to improve state-of-the-art search engine performance and how to incorporate this new input signal marked by human attention into existing neural retrieval models. In this paper, we propose GazBy (bf Gaze-based bf Bert model for document relevancbf y), a light-weight joint model that integrates human gaze fixation estimation into transformer models to predict document relevance, incorporating more nuanced information about cognitive processing into information retrieval (IR). We evaluate our model on the Text Retrieval Conference (TREC) Deep Learning (DL) 2019 and 2020 Tracks. Our experiments show encouraging results and illustrate the effective and ineffective entry points for using human gaze to help with transformer-based neural retrievers. With the rise of virtual reality (VR) and augmented reality (AR), human gaze data will become more available. We hope this work serves as a first step exploring using gaze signals in modern neural search engines.