# Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2311.00159 · HTML: https://mlsanity.com/paper/2311.00159/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling](https://mlsanity.com/paper/1611.01462/) (2016)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention](https://mlsanity.com/paper/2010.07891/) (2020)
- [Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension](https://mlsanity.com/paper/2010.06396/) (2020)
- [Entity Recognition at First Sight: Improving NER with Eye Movement Information](https://mlsanity.com/paper/1902.10068/) (2019)
- [Improving sentence compression by learning to predict gaze](https://mlsanity.com/paper/1604.03357/) (2016)
- [Synthesizing Human Gaze Feedback for Improved NLP Performance](https://mlsanity.com/paper/2302.05721/) (2023)
- [A Survey on Using Gaze Behaviour for Natural Language Processing](https://mlsanity.com/paper/2112.15471/) (2021)
- [Modeling Human Reading with Neural Attention](https://mlsanity.com/paper/1608.05604/) (2016)
- [Modeling Task Effects in Human Reading with Neural Network-based Attention](https://mlsanity.com/paper/1808.00054/) (2018)
- [Every word counts: A multilingual analysis of individual human alignment with model attention](https://mlsanity.com/paper/2210.04963/) (2022)

## Built on by (most cited first, 3 of 3)

- [Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models](https://mlsanity.com/paper/2410.01532/) (2024)
- [OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses](https://mlsanity.com/paper/2503.10927/) (2025)
- [Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions](https://mlsanity.com/paper/2504.06843/) (2025)

## Abstract (arXiv)

Humans read texts at a varying pace, while machine learning models treat each token in the same way in terms of a computational process. Therefore, we ask, does it help to make models act more like humans? In this paper, we convert this intuition into a set of novel models with fixation-guided parallel RNNs or layers and conduct various experiments on language modeling and sentiment analysis tasks to test their effectiveness, thus providing empirical validation for this intuition. Our proposed models achieve good performance on the language modeling task, considerably surpassing the baseline model. In addition, we find that, interestingly, the fixation duration predicted by neural networks bears some resemblance to humans' fixation. Without any explicit guidance, the model makes similar choices to humans. We also investigate the reasons for the differences between them, which explain why "model fixations" are often more suitable than human fixations, when used to guide language models.