mlsanityData to 14 Oct 2026

YouRefIt: Embodied Reference Understanding with Language and Gesture

arXiv 2109.03413 ↗First posted 8 Sep 2021cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

We study the understanding of embodied reference: One agent uses both language and gesture to refer to an object to another agent in a shared physical environment. Of note, this new visual task requires understanding multimodal cues with perspective-taking to identify which object is being referred to. To tackle this problem, we introduce YouRefIt, a new crowd-sourced dataset of embodied reference collected in various physical scenes; the dataset contains 4,195 unique reference clips in 432 indoor scenes. To the best of our knowledge, this is the first embodied reference dataset that allows us to study referring expressions in daily physical scenes to understand referential behavior, human communication, and human-robot interaction. We further devise two benchmarks for image-based and video-based embodied reference understanding. Comprehensive baselines and extensive experiments provide the very first result of machine perception on how the referring expressions and gestures affect the embodied reference understanding. Our results provide essential evidence that gestural cues are as critical as language cues in understanding the embodied reference.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 35 of its 75 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

35 of 75 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2017Mask R-CNNcited by 7,277
  6. 2018YOLOv3: An Incremental Improvementcited by 2,659
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  11. 2016Modeling Context in Referring Expressionscited by 1,063
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
23 more
  1. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  2. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  3. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  4. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  5. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  6. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  7. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 138
  8. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 113
  9. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  10. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  11. 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
  12. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  13. 2019CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressionscited by 73
  14. 2020Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sensecited by 62
  15. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  16. 2020Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehensioncited by 50
  17. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 47
  18. 2019Contextual Encoder-Decoder Network for Visual Saliency Predictioncited by 44
  19. 2021Learning Triadic Belief Dynamics in Nonverbal Communication from Videoscited by 17
  20. 2020Human-Robot Interaction in a Shared Augmented Reality Workspacecited by 10
  21. 2020Joint Inference of States, Robot Knowledge, and Human (False-)Beliefscited by 8
  22. 2021Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scenecited by 7
  23. 2021Individual vs. Joint Perception: a Pragmatic Model of Pointing as Communicative Smithian Helpingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 8; 2024: 6; 2025: 8; 2026: 7.

33 papers.

Sort:

By area:

  1. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 158
  2. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
  3. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  4. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  5. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  6. 2023HandMeThat: Human-Robot Communication in Physical and Social Environmentscited by 20
  7. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 19
  8. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  9. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16
  10. 2023ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understandingcited by 12
  11. 2023Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understandingcited by 11
  12. 2023STRAP: Structured Object Affordance Segmentation with Point Supervisioncited by 10
21 more
  1. 2024Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Taskscited by 8
  2. 2025Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understandingcited by 8
  3. 2022Distance-Aware Occlusion Detection with Focused Attentioncited by 6
  4. 2023ProBio: A Protocol-guided Multimodal Dataset for Molecular Biology Labcited by 6
  5. 2024Robi Butler: Multimodal Remote Interaction with a Household Robot Assistantcited by 6
  6. 2022ULN: Towards Underspecified Vision-and-Language Navigationcited by 5
  7. 2026Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Taskscited by 4
  8. 2023DetermiNet: A Large-Scale Diagnostic Dataset for Complex Visually-Grounded Referencing using Determinerscited by 3
  9. 2025Beyond Object Categories: Multi-Attribute Reference Understanding for Visual Groundingcited by 3
  10. 2025Learning to Generate Pointing Gestures in Situated Embodied Conversational Agentscited by 3
  11. 2024AD-DINO: Attention-Dynamic DINO for Distance-Aware Embodied Reference Understandingcited by 2
  12. 2025I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialoguecited by 2
  13. 2025Multimodal Data Storage and Retrieval for Embodied AI: A Surveycited by 2
  14. 2026Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Visioncited by 2
  15. 2026MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialoguecited by 2
  16. 2025A Multimodal Depth-Aware Method For Embodied Reference Understandingcited by 1
  17. 2026Investigating Multimodal Informativity under Different Partner Visibility Conditions in Video-Mediated Dialoguecited by 1
  18. 2025CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understandingcited by 0
  19. 2026Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Timecited by 0
  20. 2026PoseRefer: Pathway-Local Parameters for Semantically Grounded Reference Resolutioncited by 0
  21. 2026VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detectioncited by 0