# Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze

arXiv: https://arxiv.org/abs/2011.04592 · HTML: https://mlsanity.com/paper/2011.04592/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [Multilingual Language Models Predict Human Reading Behavior](https://mlsanity.com/paper/2104.05433/) (2021)
- [Relative Importance in Sentence Processing](https://mlsanity.com/paper/2106.03471/) (2021)
- [Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading](https://mlsanity.com/paper/2304.10784/) (2023)
- [CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals](https://mlsanity.com/paper/2106.05544/) (2021)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [Pre-Trained Language Models Augmented with Synthetic Scanpaths for Natural Language Understanding](https://mlsanity.com/paper/2310.14676/) (2023)
- [The Copenhagen Corpus of Eye Tracking Recordings from Natural Reading of Danish Texts](https://mlsanity.com/paper/2204.13311/) (2022)
- [PoTeC: A German Naturalistic Eye-tracking-while-reading Corpus](https://mlsanity.com/paper/2403.00506/) (2024)
- [EMTeC: A Corpus of Eye Movements on Machine-Generated Texts](https://mlsanity.com/paper/2408.04289/) (2024)
- [Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts](https://mlsanity.com/paper/2303.17595/) (2023)
- [DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Reading](https://mlsanity.com/paper/2310.14802/) (2023)
- [Personalized Image Descriptions from Attention Sequences](https://mlsanity.com/paper/2512.06662/) (2025)
- [Describing Images textitFast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes](https://mlsanity.com/paper/2402.01352/) (2024)
- [A look under the hood of the Interactive Deep Learning Enterprise (No-IDLE)](https://mlsanity.com/paper/2406.19054/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Human-Centric NLP or AI-Centric Illusion?: A Critical Investigation](https://mlsanity.com/paper/2412.10939/) (2024)

## Abstract (arXiv)

When speakers describe an image, they tend to look at objects before mentioning them. In this paper, we investigate such sequential cross-modal alignment by modelling the image description generation process computationally. We take as our starting point a state-of-the-art image captioning system and develop several model variants that exploit information from human gaze patterns recorded during language production. In particular, we propose the first approach to image description generation where visual processing is modelled textitsequentially. Our experiments and analyses confirm that better descriptions can be obtained by exploiting gaze-driven attention and shed light on human cognitive processes by comparing different ways of aligning the gaze modality with language production. We find that processing gaze data sequentially leads to descriptions that are better aligned to those produced by speakers, more diverse, and more natural-particularly when gaze is encoded with a dedicated recurrent component.