# Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention

arXiv: https://arxiv.org/abs/2010.07891 · HTML: https://mlsanity.com/paper/2010.07891/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 30 of 45)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [Multilingual Language Models Predict Human Reading Behavior](https://mlsanity.com/paper/2104.05433/) (2021)
- [Textual Prompt Guided Image Restoration](https://mlsanity.com/paper/2312.06162/) (2023)
- [Relative Importance in Sentence Processing](https://mlsanity.com/paper/2106.03471/) (2021)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?](https://mlsanity.com/paper/2205.10226/) (2022)
- [Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading](https://mlsanity.com/paper/2304.10784/) (2023)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [Synthesizing Human Gaze Feedback for Improved NLP Performance](https://mlsanity.com/paper/2302.05721/) (2023)
- [Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarization](https://mlsanity.com/paper/2402.14182/) (2024)
- [Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities](https://mlsanity.com/paper/2406.08379/) (2024)
- [ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts](https://mlsanity.com/paper/2310.15587/) (2023)
- [Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models](https://mlsanity.com/paper/2410.01532/) (2024)
- [CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals](https://mlsanity.com/paper/2106.05544/) (2021)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding](https://mlsanity.com/paper/2310.14676/) (2023)
- [Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention](https://mlsanity.com/paper/2310.19084/) (2023)
- [EMTeC: A Corpus of Eye Movements on Machine-Generated Texts](https://mlsanity.com/paper/2408.04289/) (2024)
- [Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts](https://mlsanity.com/paper/2303.17595/) (2023)
- [Understanding Deep Neural Networks via Linear Separability of Hidden Layers](https://mlsanity.com/paper/2307.13962/) (2023)
- [Attribute-Guided Multi-Level Attention Network for Fine-Grained Fashion Retrieval](https://mlsanity.com/paper/2301.13014/) (2022)
- [Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networks](https://mlsanity.com/paper/2311.00159/) (2023)
- [OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses](https://mlsanity.com/paper/2503.10927/) (2025)
- [A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior](https://mlsanity.com/paper/2506.19999/) (2025)
- [A Comparative Study on Textual Saliency of Styles from Eye Tracking, Annotations, and Language Models](https://mlsanity.com/paper/2212.09873/) (2022)
- [Finding Short Signals in Long Irregular Time Series with Continuous-Time Attention Policy Networks](https://mlsanity.com/paper/2302.04052/) (2023)
- [DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading](https://mlsanity.com/paper/2310.14802/) (2023)
- [Teaching Human Behavior Improves Content Understanding Abilities Of LLMs](https://mlsanity.com/paper/2405.00942/) (2024)

## Abstract (arXiv)

A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing(NLP). We propose a novel hybrid text saliency model(TSM) that, for the first time, combines a cognitive model of reading with explicit human gaze supervision in a single machine learning framework. On four different corpora we demonstrate that our hybrid TSM duration predictions are highly correlated with human gaze ground truth. We further propose a novel joint modeling approach to integrate TSM predictions into the attention layer of a network designed for a specific upstream NLP task without the need for any task-specific human gaze data. We demonstrate that our joint model outperforms the state of the art in paraphrase generation on the Quora Question Pairs corpus by more than 10% in BLEU-4 and achieves state of the art performance for sentence compression on the challenging Google Sentence Compression corpus. As such, our work introduces a practical approach for bridging between data-driven and cognitive models and demonstrates a new way to integrate human gaze-guided neural attention into NLP tasks.