Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions
Not on the map. Why?
Abstract (arXiv)
Recently, the integration of cognitive neuroscience in Natural Language Processing (NLP) has gained significant attention. This article provides a critical and timely overview of recent advancements in leveraging cognitive signals, particularly Eye-tracking (ET) signals, to enhance Language Models (LMs) and Multimodal Large Language Models (MLLMs). By incorporating user-centric cognitive signals, these approaches address key challenges, including data scarcity and the environmental costs of training large-scale models. Cognitive signals enable efficient data augmentation, faster convergence, and improved human alignment. The review emphasises the potential of ET data in tasks like Visual Question Answering (VQA) and mitigating hallucinations in MLLMs, and concludes by discussing emerging challenges and research trends.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 74 references that are in the map. First posted on arXiv 9 Apr 2025.
Builds on
42 of 74 references.
References · 42
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023A Survey of Large Language Modelscited by 2,576
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2023A Survey on Multimodal Large Language Modelscited by 909
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
- 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
30 more
- 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
- 2020Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attentioncited by 45
- 2020Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehensioncited by 29
- 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
- 2023TempSAL -- Uncovering Temporal Information for Deep Saliency Predictioncited by 25
- 2021Relative Importance in Sentence Processingcited by 22
- 2020Bridging Information-Seeking Human Gaze and Machine Reading Comprehensioncited by 20
- 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
- 2022Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?cited by 20
- 2023Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Readingcited by 19
- 2023Synthesizing Human Gaze Feedback for Improved NLP Performancecited by 16
- 2024Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Modelscited by 13
- 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 12
- 2021CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signalscited by 11
- 2021A Survey on Using Gaze Behaviour for Natural Language Processingcited by 11
- 2024Trends, Applications, and Challenges in Human Attention Modellingcited by 9
- 2024Fine-Grained Prediction of Reading Comprehension from Eye Movementscited by 9
- 2023Mapping Brains with Language Models: A Surveycited by 8
- 2023Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understandingcited by 8
- 2021TorontoCL at CMCL 2021 Shared Task: RoBERTa with Multi-Stage Fine-Tuning for Eye-Tracking Predictioncited by 7
- 2023UniAR: A Unified model for predicting human Attention and Responses on visual contentcited by 7
- 2024A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questionscited by 7
- 2023WebQAmGaze: A Multilingual Webcam Eye-Tracking-While-Reading Datasetcited by 6
- 2023Probing Large Language Models from A Human Behavioral Perspectivecited by 5
- 2024EMTeC: A Corpus of Eye Movements on Machine-Generated Textscited by 5
- 2020Learning Efficient Representations of Mouse Movements to Predict User Attentioncited by 4
- 2024TPP-Gaze: Modelling Gaze Dynamics in Space and Time with Neural Temporal Point Processescited by 4
- 2017BubbleView: an interface for crowdsourcing image importance maps and tracking visual attentioncited by 3
- 2023Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networkscited by 3
- 2024Data movement limits to frontier model trainingcited by 3
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses
- 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
- 2025Large Language Models Are Human-Like Internally
- 2026Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
- 2025Multimodal Behavioral Patterns Analysis with Eye-Tracking and LLM-Based Reasoning
- 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models
- 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
- 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models