mlsanityData to 7 Oct 2026

Video-based Human-Object Interaction Detection from Tubelet Tokens

arXiv 2206.01908 ↗First posted 4 Jun 2022cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each tubelet token is learned by a selective attention mechanism to reduce redundant spatial dependencies from others; 2) Expressiveness: each tubelet token is enabled to align with a semantic instance, i.e., an object or a human, across frames, thanks to agglomeration and linking. The effectiveness and efficiency of TUTOR are verified by extensive experiments. Results shows our method outperforms existing works by large margins, with a relative mAP gain of 16.14% on VidHOI and a 2 points gain on CAD-120 as well as a 4 × speedup.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 44 of its 61 references that are in the map. First posted on arXiv 4 Jun 2022.

Builds on

44 of 61 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2017Deformable Convolutional Networkscited by 1,481
32 more
  1. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  2. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  3. 2017Learning to Detect Human-Object Interactionscited by 238
  4. 2022VRT: A Video Restoration Transformercited by 219
  5. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  6. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  7. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  8. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  9. 2021End-to-end Temporal Action Detection with Transformercited by 150
  10. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  11. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  12. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  13. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  14. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  15. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  16. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  17. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  18. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  19. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  20. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  21. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  22. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  23. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  24. 2020Cascaded Human-Object Interaction Recognitioncited by 40
  25. 2018Interact as You Intend: Intention-Driven Human-Object Interaction Detectioncited by 30
  26. 2020Polysemy Deciphering Network for Robust Human-Object Interaction Detectioncited by 25
  27. 2021ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videoscited by 20
  28. 2022Iwin: Human-Object Interaction Detection via Transformer with Irregular Windowscited by 19
  29. 2020Knowledge-Based Video Question Answering with Unsupervised Scene Descriptionscited by 15
  30. 2020LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videoscited by 14
  31. 2020A Graph-based Interactive Reasoning for Human-Object Interaction Detectioncited by 12
  32. 2021Spatio-Temporal Interaction Graph Parsing Networks for Human-Object Interaction Recognitioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 2; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understandingcited by 18
  2. 2023Human-Object Interaction Prediction in Videos through Gaze Followingcited by 13
  3. 2023HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTscited by 12
  4. 2023Agglomerative Transformer for Human-Object Interaction Detectioncited by 8
  5. 2024Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognitioncited by 8
  6. 2023Joint Gaze-Location and Gaze-Object Detectioncited by 6
  7. 2024Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoningcited by 6
  8. 2024Jointly Modeling Spatio-Temporal Features of Tactile Signals for Action Classificationcited by 3
  9. 2025Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videoscited by 3
  10. 2026Rethinking Video Human-Object Interaction: Set Prediction over Time for Unified Detection and Anticipationcited by 2
  11. 2026IMPACT-HOI: Supervisory Control for Onset-Anchored Partial HOI Event Constructioncited by 2
  12. 2025Prototype Embedding Optimization for Human-Object Interaction Detection in Livestreamingcited by 0
1 more
  1. 2026SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understandingcited by 0