mlsanityData to 7 Oct 2026

Discovering A Variety of Objects in Spatio-Temporal Human-Object Interactions

arXiv 2211.07501 ↗First posted 14 Nov 2022cs.CV cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Spatio-temporal Human-Object Interaction (ST-HOI) detection aims at detecting HOIs from videos, which is crucial for activity understanding. In daily HOIs, humans often interact with a variety of objects, e.g., holding and touching dozens of household items in cleaning. However, existing whole body-object interaction video benchmarks usually provide limited object classes. Here, we introduce a new benchmark based on AVA: Discovering Interacted Objects (DIO) including 51 interactions and 1,000+ objects. Accordingly, an ST-HOI learning task is proposed expecting vision systems to track human actors, detect interactions and simultaneously discover interacted objects. Even though today's detectors/trackers excel in object detection/tracking tasks, they perform unsatisfied to localize diverse/unseen objects in DIO. This profoundly reveals the limitation of current vision systems and poses a great challenge. Thus, how to leverage spatio-temporal cues to address object discovery is explored, and a Hierarchical Probe Network (HPN) is devised to discover interacted objects utilizing hierarchical spatio-temporal human/context cues. In extensive experiments, HPN demonstrates impressive performance. Data and code are available at https://github.com/DirtyHarryLYL/HAKE-AVA.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 63 of its 83 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

63 of 83 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  7. 2019Objects as Pointscited by 1,172
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  9. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  12. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
38 more
  1. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  2. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  3. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 489
  4. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  5. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 424
  6. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  7. 2017Temporal Relational Reasoning in Videoscited by 324
  8. 2018Video Action Transformer Networkcited by 316
  9. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  10. 2017Relation Networks for Object Detectioncited by 305
  11. 2018CrowdHuman: A Benchmark for Detecting Human in a Crowdcited by 283
  12. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  13. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  14. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  15. 2017Learning to Detect Human-Object Interactionscited by 238
  16. 2018Videos as Space-Time Region Graphscited by 221
  17. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  18. 2019Few-Shot Object Detection with Attention-RPN and Multi-Relation Detectorcited by 206
  19. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  20. 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 199
  21. 2019Learning a Neural Solver for Multiple Object Trackingcited by 198
  22. 2018Zero-Shot Object Detectioncited by 189
  23. 2020Siamese Box Adaptive Network for Visual Trackingcited by 189
  24. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  25. 2017CityPersons: A Diverse Dataset for Pedestrian Detectioncited by 171
  26. 2016Person Re-identification in the Wildcited by 157
  27. 2021Human Pose Regression with Residual Log-likelihood Estimationcited by 131
  28. 2020Memory Enhanced Global-Local Aggregation for Video Object Detectioncited by 128
  29. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  30. 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 112
  31. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  32. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  33. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 79
  34. 2017Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependenciescited by 76
  35. 2018Object Level Visual Reasoning in Videoscited by 67
  36. 2020Asynchronous Interaction Aggregation for Action Detectioncited by 65
  37. 2017From Lifestyle Vlogs to Everyday Interactionscited by 42
  38. 2019Video Relationship Reasoning using Gated Spatio-Temporal Energy Graphcited by 41

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 0; 2026: 1.

6 papers.

  1. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  2. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  3. 2022Beyond Object Recognition: A New Benchmark towards Object Concept Learningcited by 11
  4. 2023Diagnosing Human-object Interaction Detectorscited by 11
  5. 2024Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognitioncited by 8
  6. 2026WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory and In-Context Learningcited by 4