mlsanityData to 14 Oct 2026

Synthesizing Human Gaze Feedback for Improved NLP Performance

arXiv 2302.05721 ↗First posted 11 Feb 2023cs.HC cs.CLcited by 16

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Integrating human feedback in models can improve the performance of natural language processing (NLP) models. Feedback can be either explicit (e.g. ranking used in training language models) or implicit (e.g. using human cognitive signals in the form of eyetracking). Prior eye tracking and NLP research reveal that cognitive processes, such as human scanpaths, gleaned from human gaze patterns aid in the understanding and performance of NLP models. However, the collection of real eyetracking data for NLP tasks is challenging due to the requirement of expensive and precise equipment coupled with privacy invasion issues. To address this challenge, we propose ScanTextGAN, a novel model for generating human scanpaths over text. We show that ScanTextGAN-generated scanpaths can approximate meaningful cognitive signals in human gaze patterns. We include synthetically generated scanpaths in four popular NLP tasks spanning six different datasets as proof of concept and show that the models augmented with generated scanpaths improve the performance of all downstream NLP tasks.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 58 references that are in the map. First posted on arXiv 11 Feb 2023.

Builds on

29 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  7. 2019Salient Object Detection in the Deep Learning Era: An In-Depth Surveycited by 164
  8. 2020Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attentioncited by 45
  9. 2018PathGAN: Visual Scanpath Prediction with Generative Adversarial Networkscited by 32
  10. 2021Multilingual Language Models Predict Human Reading Behaviorcited by 29
  11. 2021Human-Aided Saliency Maps Improve Generalization of Deep Learningcited by 27
  12. 2019Entity Recognition at First Sight: Improving NER with Eye Movement Informationcited by 26
16 more
  1. 2016Gaze Embeddings for Zero-Shot Image Classificationcited by 25
  2. 2016Improving sentence compression by learning to predict gazecited by 24
  3. 2021ScanGAN360: A Generative Model of Realistic Scanpaths for 360circ Imagescited by 23
  4. 2021State-of-the-Art in Human Scanpath Predictioncited by 22
  5. 2017Leveraging Cognitive Features for Sentiment Analysiscited by 19
  6. 2018Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehensioncited by 19
  7. 2017Supervising Neural Attention Models for Video Captioning by Human Gaze Datacited by 18
  8. 2019Advancing NLP with Cognitive Language Processing Signalscited by 18
  9. 2021CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signalscited by 11
  10. 2021A Survey on Using Gaze Behaviour for Natural Language Processingcited by 11
  11. 2016Modeling Human Reading with Neural Attentioncited by 10
  12. 2016Keystroke dynamics as signal for shallow syntactic parsingcited by 8
  13. 2017Harnessing Cognitive Features for Sarcasm Detectioncited by 8
  14. 2019Towards Making a Dependency Parser Seecited by 5
  15. 2016Learning Sentence Representation with Guidance of Human Attentioncited by 2
  16. 2019Automatic learner summary assessment for reading comprehensioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 6; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 12
  2. 2023Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behaviorcited by 9
  3. 2023Long-Term Ad Memorability: Understanding & Generating Memorable Adscited by 9
  4. 2024Trends, Applications, and Challenges in Human Attention Modellingcited by 9
  5. 2024Keystroke Dynamics Against Academic Dishonesty in the Age of LLMscited by 6
  6. 2024EMTeC: A Corpus of Eye Movements on Machine-Generated Textscited by 5
  7. 2023Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproductscited by 4
  8. 2023Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networkscited by 3
  9. 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responsescited by 3
  10. 2025LLM-Assisted Cheating Detection in Korean Language via Keystrokescited by 3
  11. 2024Teaching Human Behavior Improves Content Understanding Abilities Of LLMscited by 2
  12. 2025Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directionscited by 2
4 more
  1. 2025Machine-assisted writing evaluation: Exploring pre-trained language models in analyzing argumentative movescited by 0
  2. 2025A Comparative Study of Scanpath Models in Graph-Based Visualizationcited by 0
  3. 2025Decoding Open-Ended Information Seeking Goals from Eye Movements in Readingcited by 0
  4. 2026Eyettention II: A Dual-Sequence Architecture for Modeling Fixation Location, Within-Word Landing Position, and Fixation Duration in Readingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts
  2. 2026Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention
  3. 2025Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction
  4. 2026Learning to Refer from Estimated Listener Gaze
  5. 2026DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction
  6. 2026Gaze Heads: How VLMs Look at What They Describe
  7. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  8. 2024NL-Eye: Abductive NLI for Images