# CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals

arXiv: https://arxiv.org/abs/2106.05544 · HTML: https://mlsanity.com/paper/2106.05544/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks](https://mlsanity.com/paper/1506.05751/) (2015)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Multi-task Sequence to Sequence Learning](https://mlsanity.com/paper/1511.06114/) (2015)
- [Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification](https://mlsanity.com/paper/1606.01614/) (2016)
- [Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention](https://mlsanity.com/paper/2010.07891/) (2020)
- [Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension](https://mlsanity.com/paper/2010.06396/) (2020)
- [Attentive Pooling Networks](https://mlsanity.com/paper/1602.03609/) (2016)
- [Entity Recognition at First Sight: Improving NER with Eye Movement Information](https://mlsanity.com/paper/1902.10068/) (2019)
- [Leveraging Cognitive Features for Sentiment Analysis](https://mlsanity.com/paper/1701.05581/) (2017)
- [Advancing NLP with Cognitive Language Processing Signals](https://mlsanity.com/paper/1904.02682/) (2019)
- [Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze](https://mlsanity.com/paper/2011.04592/) (2020)
- [Towards Making a Dependency Parser See](https://mlsanity.com/paper/1909.01053/) (2019)
- [Human brain activity for machine attention](https://mlsanity.com/paper/2006.05113/) (2020)
- [Adversarial Learning for Chinese NER from Crowd Annotations](https://mlsanity.com/paper/1801.05147/) (2018)

## Built on by (most cited first, 11 of 11)

- [Synthesizing Human Gaze Feedback for Improved NLP Performance](https://mlsanity.com/paper/2302.05721/) (2023)
- [Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models](https://mlsanity.com/paper/2410.01532/) (2024)
- [Do Large Language Models Mirror Cognitive Language Processing?](https://mlsanity.com/paper/2402.18023/) (2024)
- [OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses](https://mlsanity.com/paper/2503.10927/) (2025)
- [DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading](https://mlsanity.com/paper/2310.14802/) (2023)
- [Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions](https://mlsanity.com/paper/2504.06843/) (2025)
- [Bridging between Cognitive Processing Signals and Linguistic Features via a Unified Attentional Network](https://mlsanity.com/paper/2112.08831/) (2021)
- [Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning](https://mlsanity.com/paper/2310.14785/) (2023)
- [Describing Images textitFast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes](https://mlsanity.com/paper/2402.01352/) (2024)
- [Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence](https://mlsanity.com/paper/2608.16975/) (2026)
- [Eye-Tracking-while-Reading: A Living Survey of Datasets with Open Library Support](https://mlsanity.com/paper/2602.19598/) (2026)

## Abstract (arXiv)

Most previous studies integrate cognitive language processing signals (e.g., eye-tracking or EEG data) into neural models of natural language processing (NLP) just by directly concatenating word embeddings with cognitive features, ignoring the gap between the two modalities (i.e., textual vs. cognitive) and noise in cognitive features. In this paper, we propose a CogAlign approach to these issues, which learns to align textual neural representations to cognitive features. In CogAlign, we use a shared encoder equipped with a modality discriminator to alternatively encode textual and cognitive inputs to capture their differences and commonalities. Additionally, a text-aware attention mechanism is proposed to detect task-related information and to avoid using noise in cognitive features. Experimental results on three NLP tasks, namely named entity recognition, sentiment analysis and relation extraction, show that CogAlign achieves significant improvements with multiple cognitive features over state-of-the-art models on public datasets. Moreover, our model is able to transfer cognitive information to other datasets that do not have any cognitive processing signals.