mlsanityData to 14 Oct 2026

The Overlooked Classifier in Human-Object Interaction Recognition

arXiv 2112.06392 ↗First posted 13 Dec 2021cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) recognition is challenging due to two factors: (1) significant imbalance across classes and (2) requiring multiple labels per image. This paper shows that these two challenges can be effectively addressed by improving the classifier with the backbone architecture untouched. Firstly, we encode the semantic correlation among classes into the classification head by initializing the weights with language embeddings of HOIs. As a result, the performance is boosted significantly, especially for the few-shot subset. Secondly, we propose a new loss named LSE-Sign to enhance multi-label learning on a long-tailed dataset. Our simple yet effective method enables detection-free HOI classification, outperforming the state-of-the-arts that require object detection and human pose by a clear margin. Moreover, we transfer the classification model to instance-level HOI detection by connecting it with an off-the-shelf object detector. We achieve state-of-the-art without additional fine-tuning.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 13 Dec 2021.

Builds on

43 of 54 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
31 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  3. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  4. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 403
  5. 2017Learning to Detect Human-Object Interactionscited by 240
  6. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  7. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  8. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  9. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  10. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  11. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  12. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  13. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  14. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 127
  15. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  16. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  17. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  18. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  19. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  20. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  21. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 80
  22. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  23. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  24. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  25. 2017Attentional Pooling for Action Recognitioncited by 54
  26. 2019Detecting Human-Object Interactions via Functional Generalizationcited by 53
  27. 2015Contextual Action Recognition with R*CNNcited by 34
  28. 2018Pairwise Body-Part Attention for Recognizing Human-Object Interactionscited by 34
  29. 2019HAKE: Human Activity Knowledge Enginecited by 32
  30. 2014Feedforward semantic segmentation with zoom-out featurescited by 21
  31. 2016Learning Models for Actions and Person-Object Interactions with Transfer to Question Answeringcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 1; 2025: 0; 2026: 2.

8 papers.

  1. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  2. 2023Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Modelcited by 22
  3. 2024CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generationcited by 5
  4. 2023Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisorcited by 4
  5. 2022UV R-CNN: Stable and Efficient Dense Human Pose Estimationcited by 0
  6. 2023Mining Conditional Part Semantics with Occluded Extrapolation for Human-Object Interaction Detectioncited by 0
  7. 2026CoViT: Instance-Correspondence Contrastive Learning for Vision Transformercited by 0
  8. 2026Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interactioncited by 0