mlsanityData to 14 Oct 2026

Look Hear: Gaze Prediction for Speech-directed Human Attention

arXiv 2407.19605 ↗First posted 28 Jul 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

For computer systems to effectively interact with humans using spoken language, they need to understand how the words being generated affect the users' moment-by-moment attention. Our study focuses on the incremental prediction of attention as a person is seeing an image and hearing a referring expression defining the object in the scene that should be fixated by gaze. To predict the gaze scanpaths in this incremental object referral task, we developed the Attention in Referral Transformer model or ART, which predicts the human fixations spurred by each word in a referring expression. ART uses a multimodal transformer encoder to jointly learn gaze behavior and its underlying grounding tasks, and an autoregressive transformer decoder to predict, for each word, a variable number of fixations based on fixation history. To train ART, we created RefCOCO-Gaze, a large-scale dataset of 19,738 human gaze scanpaths, corresponding to 2,094 unique image-expression pairs, from 220 participants performing our referral task. In our quantitative and qualitative analyses, ART not only outperforms existing methods in scanpath prediction, but also appears to capture several human attention patterns, such as waiting, scanning, and verification.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 45 of its 101 references that are in the map. First posted on arXiv 28 Jul 2024.

Builds on

45 of 101 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,368
33 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  3. 2016Modeling Context in Referring Expressionscited by 1,063
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  6. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  10. 2019Connecting Vision and Language with Localized Narrativescited by 226
  11. 2018A Simple Method for Commonsense Reasoningcited by 223
  12. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  13. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  14. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 140
  15. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  16. 2014From Captions to Visual Concepts and Backcited by 109
  17. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
  18. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  19. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  20. 2020Norm-Based Curriculum Learning for Neural Machine Translationcited by 60
  21. 2018Real-Time Referring Expression Comprehension by Single-Stage Grounding Networkcited by 56
  22. 2020Predicting Goal-directed Human Attention Using Inverse Reinforcement Learningcited by 53
  23. 2020Integrating Human Gaze into Attention for Egocentric Activity Recognitioncited by 42
  24. 2018Object Referring in Videos with Language and Human Gazecited by 40
  25. 2023Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attentioncited by 38
  26. 2020AiR: Attention with Reasoning Capabilitycited by 28
  27. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  28. 2022TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillationcited by 25
  29. 2023Unifying Top-down and Bottom-up Scanpath Prediction Using Transformerscited by 22
  30. 2022Target-absent Human Attentioncited by 20
  31. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16
  32. 2020Predicting Goal-directed Attention Control Using Inverse-Reinforcement Learningcited by 8
  33. 2017Object Referring in Visual Scene with Spoken Languagecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Predictioncited by 7
  2. 2025Reading Recognition in the Wildcited by 6
  3. 2025Few-shot Personalized Scanpath Predictioncited by 5
  4. 2026Gaze-to-text Generation: Beyond Categorical Decoding of Human Attentioncited by 0
  5. 2026Learning to Refer from Estimated Listener Gazecited by 0