mlsanityData to 7 Oct 2026

Object Referring in Videos with Language and Human Gaze

arXiv 1801.01582 ↗First posted 4 Jan 2018cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

We investigate the problem of object referring (OR) i.e. to localize a target object in a visual scene coming with a language description. Humans perceive the world more as continued video snippets than as static images, and describe objects not only by their appearance, but also by their spatio-temporal context and motion features. Humans also gaze at the object when they issue a referring expression. Existing works for OR mostly focus on static images only, which fall short in providing many such cues. This paper addresses OR in videos with language and human gaze. To that end, we present a new video dataset for OR, with 30, 000 objects over 5, 000 stereo video sequences annotated for their descriptions and gaze. We further propose a novel network model for OR in videos, by integrating appearance, motion, gaze, and spatio-temporal context into one network. Experimental results show that our method effectively utilizes motion cues, human gaze, and spatio-temporal context. Our method outperforms previousOR methods. For dataset and code, please refer https://people.ee.ethz.ch/~arunv/ORGaze.html.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 10; 2024: 10; 2025: 4; 2026: 4.

40 papers.

Sort:

By area:

  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  2. 2023Language Prompt for Autonomous Drivingcited by 110
  3. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  4. 2023Referring Multi-Object Trackingcited by 91
  5. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  6. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Modelscited by 75
  7. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  8. 2022Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Groundingcited by 40
  9. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  10. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
  11. 2024Bootstrapping Referring Multi-Object Trackingcited by 21
  12. 2022Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?cited by 20
28 more
  1. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  2. 2022RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classificationcited by 19
  3. 2022End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Groundingcited by 19
  4. 2025STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Drivingcited by 17
  5. 2024Referring Atomic Video Action Recognitioncited by 16
  6. 2023Look, Remember and Reason: Grounded reasoning in videos with language modelscited by 13
  7. 2025RefAV: Towards Planning-Centric Scenario Miningcited by 13
  8. 2024Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimationcited by 12
  9. 20244-LEGS: 4D Language Embedded Gaussian Splattingcited by 12
  10. 2022Correspondence Matters for Video Referring Expression Comprehensioncited by 10
  11. 2024MLS-Track: Multilevel Semantic Interaction in RMOTcited by 7
  12. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  13. 2021Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?cited by 6
  14. 2023Multi-Modal Gaze Following in Conversational Scenarioscited by 6
  15. 2024Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentationcited by 6
  16. 2021Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Datacited by 5
  17. 2024Look Hear: Gaze Prediction for Speech-directed Human Attentioncited by 5
  18. 2022Video Referring Expression Comprehension via Transformer with Content-aware Querycited by 4
  19. 2023Video Referring Expression Comprehension via Transformer with Content-conditioned Querycited by 4
  20. 2025ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Modelcited by 4
  21. 2022Gaussian Kernel-based Cross Modal Network for Spatio-Temporal Video Groundingcited by 3
  22. 2026Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoningcited by 2
  23. 2021Neural Variational Learning for Grounded Language Acquisitioncited by 1
  24. 2022Where and What: Driver Attention-based Object Detectioncited by 1
  25. 2025Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Viewscited by 1
  26. 2026Talk2DM: Enabling Natural Language Querying and Commonsense Reasoning for Vehicle-Road-Cloud Integrated Dynamic Maps with Large Language Modelscited by 0
  27. 2026STORM: End-to-End Referring Multi-Object Tracking in Videoscited by 0
  28. 2026Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verificationcited by 0