mlsanityData to 7 Oct 2026

HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision

arXiv 2404.09933 ↗First posted 15 Apr 2024cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Vision Language Models (VLMs) are now the de facto state-of-the-art for a number of tasks including visual question answering, recognising objects, and spatial referral. In this work, we propose the HOI-Ref task for egocentric images that aims to understand interactions between hands and objects using VLMs. To enable HOI-Ref, we curate the HOI-QA dataset that consists of 3.9M question-answer pairs for training and evaluating VLMs. HOI-QA includes questions relating to locating hands, objects, and critically their interactions (e.g. referring to the object being manipulated by the hand). We train the first VLM for HOI-Ref on this dataset and call it VLM4HOI. Our results demonstrate that VLMs trained for referral on third person images fail to recognise and refer hands and objects in egocentric images. When fine-tuned on our egocentric HOI-QA dataset, performance improves by 27.9% for referring hands and objects, and by 26.7% for referring interactions.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 67 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

49 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition18 Jan

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
36 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  14. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  15. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  16. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  18. 2020Rescaling Egocentric Visioncited by 529
  19. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  20. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  21. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  22. 2022Egocentric Video-Language Pretrainingcited by 231
  23. 2022Learning Video Representations from Large Language Modelscited by 204
  24. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  25. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  26. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  27. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  28. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 102
  29. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  30. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  31. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 50
  32. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42
  33. 2020Detecting Hands and Recognizing Physical Contact in the Wildcited by 38
  34. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  35. 2023RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4Dcited by 21
  36. 2019Contextual Attention for Hand Detection in the Wildcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 2.

9 papers.

  1. 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 62
  2. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  3. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  4. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  5. 2026FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understandingcited by 4
  6. 2025HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamicscited by 2
  7. 2025THOR: Thermal-guided Hand-Object Reasoning via Adaptive Vision Samplingcited by 0
  8. 2025Learning Egocentric In-Hand Object Segmentation through Weak Supervision from Human Narrationscited by 0
  9. 2026Object Referring-Guided Scanpath Prediction with Perception-Enhanced Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI
  2. 2026Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision
  3. 2026EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning
  4. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis
  5. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Prediction
  6. 2026Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering
  7. 2026Towards Unconstrained Human-Object Interaction
  8. 2026HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models