mlsanityData to 14 Oct 2026

Interacted Object Grounding in Spatio-Temporal Human-Object Interactions

arXiv 2412.19542 ↗First posted 27 Dec 2024cs.CV cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Spatio-temporal Human-Object Interaction (ST-HOI) understanding aims at detecting HOIs from videos, which is crucial for activity understanding. However, existing whole-body-object interaction video benchmarks overlook the truth that open-world objects are diverse, that is, they usually provide limited and predefined object classes. Therefore, we introduce a new open-world benchmark: Grounding Interacted Objects (GIO) including 1,098 interacted objects class and 290K interacted object boxes annotation. Accordingly, an object grounding task is proposed expecting vision systems to discover interacted objects. Even though today's detectors and grounding methods have succeeded greatly, they perform unsatisfactorily in localizing diverse and rare objects in GIO. This profoundly reveals the limitations of current vision systems and poses a great challenge. Thus, we explore leveraging spatio-temporal cues to address object grounding and propose a 4D question-answering framework (4D-QA) to discover interacted objects from diverse videos. Our method demonstrates significant superiority in extensive experiments compared to current baselines. Data and code will be publicly available at https://github.com/DirtyHarryLYL/HAKE-AVA.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

40 of 56 references.

References · 40

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  9. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 692
  10. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  11. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  12. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
28 more
  1. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  2. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 429
  3. 2017Relation Networks for Object Detectioncited by 305
  4. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  5. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  6. 2017Learning to Detect Human-Object Interactionscited by 240
  7. 2018Videos as Space-Time Region Graphscited by 221
  8. 2019Learning a Neural Solver for Multiple Object Trackingcited by 198
  9. 2020Siamese Box Adaptive Network for Visual Trackingcited by 189
  10. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  11. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  12. 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 114
  13. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  14. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  15. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  16. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 80
  17. 2017Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependenciescited by 76
  18. 2024Context-Guided Spatio-Temporal Video Groundingcited by 46
  19. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  20. 2019Explainable Video Action Reasoning via Prior Knowledge and State Transitionscited by 33
  21. 2019HAKE: Human Activity Knowledge Enginecited by 32
  22. 2020Video Object Grounding using Semantic Roles in Language Descriptioncited by 25
  23. 2017Temporal Dynamic Graph LSTM for Action-driven Video Object Detectioncited by 23
  24. 2021ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videoscited by 20
  25. 2021Tracking People by Predicting 3D Appearance, Location & Posecited by 15
  26. 2023Human-Object Interaction Prediction in Videos through Gaze Followingcited by 13
  27. 2019Activity Driven Weakly Supervised Object Detectioncited by 9
  28. 2019Tell Me What They're Holding: Weakly-supervised Object Detection with Transferable Knowledge from Human-object Interactioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  2. 2025Human Body Restoration with One-Step Diffusion Model and A New Benchmarkcited by 5
  3. 2025HAODiff: Human-Aware One-Step Diffusion via Dual-Prompt Guidancecited by 5
  4. 2025FideDiff: Efficient Diffusion Model for High-Fidelity Image Motion Deblurringcited by 4
  5. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  6. 2026HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaborationcited by 0