# Multilingual Language Models Predict Human Reading Behavior

arXiv: https://arxiv.org/abs/2104.05433 · HTML: https://mlsanity.com/paper/2104.05433/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 66 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language](https://mlsanity.com/paper/1905.07213/) (2019)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation](https://mlsanity.com/paper/1912.00903/) (2019)
- [How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking](https://mlsanity.com/paper/2004.14992/) (2020)
- [Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention](https://mlsanity.com/paper/2010.07891/) (2020)
- [Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains](https://mlsanity.com/paper/1906.01539/) (2019)
- [Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity](https://mlsanity.com/paper/2003.04866/) (2020)
- [Entity Recognition at First Sight: Improving NER with Eye Movement Information](https://mlsanity.com/paper/1902.10068/) (2019)
- [Improving sentence compression by learning to predict gaze](https://mlsanity.com/paper/1604.03357/) (2016)
- [Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling](https://mlsanity.com/paper/2009.03954/) (2020)
- [Investigating representations of verb bias in neural language models](https://mlsanity.com/paper/2010.02375/) (2020)
- [Bridging Information-Seeking Human Gaze and Machine Reading Comprehension](https://mlsanity.com/paper/2009.14780/) (2020)
- [Advancing NLP with Cognitive Language Processing Signals](https://mlsanity.com/paper/1904.02682/) (2019)
- [How well does surprisal explain N400 amplitude under different experimental conditions?](https://mlsanity.com/paper/2010.04844/) (2020)
- [Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze](https://mlsanity.com/paper/2011.04592/) (2020)
- [Modeling Human Reading with Neural Attention](https://mlsanity.com/paper/1608.05604/) (2016)

## Built on by (most cited first, 29 of 29)

- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail](https://mlsanity.com/paper/2110.08300/) (2021)
- [Relative Importance in Sentence Processing](https://mlsanity.com/paper/2106.03471/) (2021)
- [Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?](https://mlsanity.com/paper/2205.10226/) (2022)
- [Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading](https://mlsanity.com/paper/2304.10784/) (2023)
- [Synthesizing Human Gaze Feedback for Improved NLP Performance](https://mlsanity.com/paper/2302.05721/) (2023)
- [Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models](https://mlsanity.com/paper/2311.09194/) (2023)
- [ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts](https://mlsanity.com/paper/2310.15587/) (2023)
- [From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes](https://mlsanity.com/paper/2410.22906/) (2024)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Measuring the Impact of (Psycho-)Linguistic and Readability Features and Their Spill Over Effects on the Prediction of Eye Movement Patterns](https://mlsanity.com/paper/2203.08085/) (2022)
- [Large Language Models for Psycholinguistic Plausibility Pretesting](https://mlsanity.com/paper/2402.05455/) (2024)
- [PoTeC: A German Naturalistic Eye-tracking-while-reading Corpus](https://mlsanity.com/paper/2403.00506/) (2024)
- [EMTeC: A Corpus of Eye Movements on Machine-Generated Texts](https://mlsanity.com/paper/2408.04289/) (2024)
- [Every word counts: A multilingual analysis of individual human alignment with model attention](https://mlsanity.com/paper/2210.04963/) (2022)
- [OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses](https://mlsanity.com/paper/2503.10927/) (2025)
- [Cross-Lingual Transfer of Cognitive Processing Complexity](https://mlsanity.com/paper/2302.12695/) (2023)
- [Reverse-Engineering the Reader](https://mlsanity.com/paper/2410.13086/) (2024)
- [Describing Images textitFast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes](https://mlsanity.com/paper/2402.01352/) (2024)
- [Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not](https://mlsanity.com/paper/2604.04825/) (2026)
- [NU HLT at CMCL 2022 Shared Task: Multilingual and Crosslingual Prediction of Human Reading Behavior in Universal Language Space](https://mlsanity.com/paper/2202.10855/) (2022)
- [Team 'UFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models](https://mlsanity.com/paper/2204.04998/) (2022)
- [Text Difficulty Study: Do machines behave the same as humans regarding text difficulty?](https://mlsanity.com/paper/2208.14509/) (2022)
- [Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing](https://mlsanity.com/paper/2310.18229/) (2023)
- [Evaluating Webcam-based Gaze Data as an Alternative for Human Rationale Annotations](https://mlsanity.com/paper/2402.19133/) (2024)
- [Decoding Open-Ended Information Seeking Goals from Eye Movements in Reading](https://mlsanity.com/paper/2505.02872/) (2025)
- [Eyettention II: A Dual-Sequence Architecture for Modeling Fixation Location, Within-Word Landing Position, and Fixation Duration in Reading](https://mlsanity.com/paper/2606.01964/) (2026)
- [Do Cantonese-Adapted Language Models Better Predict Cantonese Reading? A Cross-Model Eye-Tracking Evaluation](https://mlsanity.com/paper/2609.02163/) (2026)

## Abstract (arXiv)

We analyze if large language models are able to predict patterns of human reading behavior. We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing on Dutch, English, German, and Russian texts. This results in accurate models of human reading behavior, which indicates that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.