mlsanityData to 14 Oct 2026

Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions

arXiv 2110.03562 ↗First posted 7 Oct 2021cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce the task of weakly supervised learning for detecting human and object interactions in videos. Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object. To address these challenges, we introduce a contrastive weakly supervised training loss that aims to jointly associate spatiotemporal regions in a video with an action and object vocabulary and encourage temporal continuity of the visual appearance of moving objects as a form of self-supervision. To train our model, we introduce a dataset comprising over 6.5k videos with human-object interaction annotations that have been semi-automatically curated from sentence captions associated with the videos. We demonstrate improved performance over weakly supervised baselines adapted to our task on our video dataset.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 42 of its 60 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

42 of 60 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  11. 2017Dense-Captioning Events in Videoscited by 793
  12. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
30 more
  1. 2017Localizing Moments in Video with Natural Languagecited by 663
  2. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  3. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  4. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 575
  5. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  6. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  7. 2017Learning to Detect Human-Object Interactionscited by 240
  8. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  9. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  10. 2015A Dataset for Movie Descriptioncited by 195
  11. 2015Visual Semantic Role Labelingcited by 190
  12. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  13. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  14. 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
  15. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  16. 2017Visual Relationship Detection with Internal and External Linguistic Knowledge Distillationcited by 61
  17. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  18. 2017From Lifestyle Vlogs to Everyday Interactionscited by 42
  19. 2019Video Relationship Reasoning using Gated Spatio-Temporal Energy Graphcited by 41
  20. 2018Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interactioncited by 40
  21. 2017Weakly-supervised learning of visual relationscited by 36
  22. 2016Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cuescited by 33
  23. 2019On Exploring Undetermined Relationships for Visual Relationship Detectioncited by 26
  24. 2017PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCNcited by 25
  25. 2017Temporal Dynamic Graph LSTM for Action-driven Video Object Detectioncited by 23
  26. 2020Visual Relation Grounding in Videoscited by 19
  27. 2018Improving Visual Relationship Detection using Semantic Modeling of Scene Descriptionscited by 13
  28. 2018Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Featurescited by 11
  29. 2020Novel Human-Object Interaction Detection via Adversarial Domain Generalizationcited by 3
  30. 2018Visual Relationship Detection Based on Guided Proposals and Semantic Knowledge Distillationcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 0; 2025: 4.

8 papers.

  1. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 37
  2. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  3. 2025HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generationcited by 26
  4. 2023What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructionscited by 10
  5. 2025MATRIX: Mask Track Alignment for Interaction-aware Video Generationcited by 4
  6. 2025Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videoscited by 3
  7. 2023Modelling Spatio-Temporal Interactions For Compositional Action Recognitioncited by 1
  8. 2025VideoGEM: Training-free Action Grounding in Videoscited by 0