mlsanityData to 14 Oct 2026

Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

arXiv 2410.01532 ↗First posted 2 Oct 2024cs.CL cs.AI cs.CVcited by 12

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Advancements in Natural Language Processing (NLP), have led to the emergence of Large Language Models (LLMs) such as GPT, Llama, Claude, and Gemini, which excel across a range of tasks but require extensive fine-tuning to align their outputs with human expectations. A widely used method for achieving this alignment is Reinforcement Learning from Human Feedback (RLHF), which, despite its success, faces challenges in accurately modelling human preferences. In this paper, we introduce GazeReward, a novel framework that integrates implicit feedback -- and specifically eye-tracking (ET) data -- into the Reward Model (RM). In addition, we explore how ET-based features can provide insights into user preferences. Through ablation studies we test our framework with different integration methods, LLMs, and ET generator models, demonstrating that our approach significantly improves the accuracy of the RM on established human preference datasets. This work advances the ongoing discussion on optimizing AI alignment with human values, exploring the potential of cognitive data for shaping future NLP research.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 95 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

53 of 95 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment24 Jul
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2016Concrete Problems in AI Safetycited by 1,642
36 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  5. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  7. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  11. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  12. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  13. 2023Self-Alignment with Instruction Backtranslationcited by 151
  14. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  15. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  16. 2024Self-Taught Evaluatorscited by 72
  17. 2019ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotationcited by 66
  18. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  19. 2020Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attentioncited by 45
  20. 2020Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehensioncited by 29
  21. 2019Entity Recognition at First Sight: Improving NER with Eye Movement Informationcited by 26
  22. 2016Improving sentence compression by learning to predict gazecited by 24
  23. 2021Relative Importance in Sentence Processingcited by 22
  24. 2017Leveraging Cognitive Features for Sentiment Analysiscited by 19
  25. 2023Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Readingcited by 19
  26. 2019Advancing NLP with Cognitive Language Processing Signalscited by 18
  27. 2023Synthesizing Human Gaze Feedback for Improved NLP Performancecited by 16
  28. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedbackcited by 15
  29. 2023ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Textscited by 14
  30. 2021CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signalscited by 11
  31. 2021A Survey on Using Gaze Behaviour for Natural Language Processingcited by 11
  32. 2018Eyes are the Windows to the Soul: Predicting the Rating of Text Quality Using Gaze Behaviourcited by 9
  33. 2024EyeTrans: Merging Human and Machine Attention for Neural Code Summarizationcited by 9
  34. 2023Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understandingcited by 8
  35. 2021TorontoCL at CMCL 2021 Shared Task: RoBERTa with Multi-Stage Fine-Tuning for Eye-Tracking Predictioncited by 7
  36. 2023Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responsescited by 3
  2. 2026Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Controlcited by 3
  3. 2025Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directionscited by 2
  4. 2025Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Spancited by 2
  5. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  6. 2025Enhancing RLHF with Human Gaze Modelingcited by 1
  7. 2026Gaze patterns predict preference and confidence in pairwise AI image evaluationcited by 1
  8. 2025Decoding Open-Ended Information Seeking Goals from Eye Movements in Readingcited by 0
  9. 2025Plural Voices, Single Agent: Towards Inclusive AI in Multi-User Domestic Spacescited by 0
  10. 2026Eye-Tracking-while-Reading: A Living Survey of Datasets with Open Library Supportcited by 0
  11. 2026Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Userscited by 0
  12. 2026Learning to Refer from Estimated Listener Gazecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  2. 2024Direct Language Model Alignment from Online AI Feedback
  3. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
  4. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  5. 2023Preference Ranking Optimization for Human Alignment
  6. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  7. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  8. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment