# Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2010.06396 · HTML: https://mlsanity.com/paper/2010.06396/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 29 of 29)

- [Relative Importance in Sentence Processing](https://mlsanity.com/paper/2106.03471/) (2021)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?](https://mlsanity.com/paper/2205.10226/) (2022)
- [Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading](https://mlsanity.com/paper/2304.10784/) (2023)
- [VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13116/) (2021)
- [ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts](https://mlsanity.com/paper/2310.15587/) (2023)
- [Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models](https://mlsanity.com/paper/2410.01532/) (2024)
- [CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals](https://mlsanity.com/paper/2106.05544/) (2021)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [LATTE: LSTM Self-Attention based Anomaly Detection in Embedded Automotive Platforms](https://mlsanity.com/paper/2107.05561/) (2021)
- [The Copenhagen Corpus of Eye Tracking Recordings from Natural Reading of Danish Texts](https://mlsanity.com/paper/2204.13311/) (2022)
- [WebQAmGaze: A Multilingual Webcam Eye-Tracking-While-Reading Dataset](https://mlsanity.com/paper/2303.17876/) (2023)
- [PoTeC: A German Naturalistic Eye-tracking-while-reading Corpus](https://mlsanity.com/paper/2403.00506/) (2024)
- [EMTeC: A Corpus of Eye Movements on Machine-Generated Texts](https://mlsanity.com/paper/2408.04289/) (2024)
- [Every word counts: A multilingual analysis of individual human alignment with model attention](https://mlsanity.com/paper/2210.04963/) (2022)
- [Human Attention during Goal-directed Reading Comprehension Relies on Task Optimization](https://mlsanity.com/paper/2107.05799/) (2021)
- [Visual Attention Emerges from Recurrent Sparse Reconstruction](https://mlsanity.com/paper/2204.10962/) (2022)
- [Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networks](https://mlsanity.com/paper/2311.00159/) (2023)
- [OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses](https://mlsanity.com/paper/2503.10927/) (2025)
- [A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior](https://mlsanity.com/paper/2506.19999/) (2025)
- [A Comparative Study on Textual Saliency of Styles from Eye Tracking, Annotations, and Language Models](https://mlsanity.com/paper/2212.09873/) (2022)
- [Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions](https://mlsanity.com/paper/2504.06843/) (2025)
- [BI AVAN: Brain inspired Adversarial Visual Attention Network](https://mlsanity.com/paper/2210.15790/) (2022)
- [Perturbation-based Self-supervised Attention for Attention Bias in Text Classification](https://mlsanity.com/paper/2305.15684/) (2023)
- [Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models](https://mlsanity.com/paper/2602.16608/) (2026)
- [Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision-Language Models](https://mlsanity.com/paper/2601.18065/) (2026)
- [Eye-Tracking-while-Reading: A Living Survey of Datasets with Open Library Support](https://mlsanity.com/paper/2602.19598/) (2026)
- [Eyettention II: A Dual-Sequence Architecture for Modeling Fixation Location, Within-Word Landing Position, and Fixation Duration in Reading](https://mlsanity.com/paper/2606.01964/) (2026)
- [Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans](https://mlsanity.com/paper/2608.16514/) (2026)

## Abstract (arXiv)

While neural networks with attention mechanisms have achieved superior performance on many natural language processing tasks, it remains unclear to which extent learned attention resembles human visual attention. In this paper, we propose a new method that leverages eye-tracking data to investigate the relationship between human visual attention and neural attention in machine reading comprehension. To this end, we introduce a novel 23 participant eye tracking dataset - MQA-RC, in which participants read movie plots and answered pre-defined questions. We compare state of the art networks based on long short-term memory (LSTM), convolutional neural models (CNN) and XLNet Transformer architectures. We find that higher similarity to human attention and performance significantly correlates to the LSTM and CNN models. However, we show this relationship does not hold true for the XLNet models -- despite the fact that the XLNet performs best on this challenging task. Our results suggest that different architectures seem to learn rather different neural attention strategies and similarity of neural to human attention does not guarantee best performance.