mlsanityData to 7 Oct 2026

Improving Human-Object Interaction Detection via Virtual Image Learning

arXiv 2308.02606 ↗First posted 4 Aug 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection aims to understand the interactions between humans and objects, which plays a curtail role in high-level semantic understanding tasks. However, most works pursue designing better architectures to learn overall features more efficiently, while ignoring the long-tail nature of interaction-object pair categories. In this paper, we propose to alleviate the impact of such an unbalanced distribution via Virtual Image Leaning (VIL). Firstly, a novel label-to-image approach, Multiple Steps Image Creation (MUSIC), is proposed to create a high-quality dataset that has a consistent distribution with real images. In this stage, virtual images are generated based on prompts with specific characterizations and selected by multi-filtering processes. Secondly, we use both virtual and real images to train the model with the teacher-student framework. Considering the initial labels of some virtual images are inaccurate and inadequate, we devise an Adaptive Matching-and-Filtering (AMF) module to construct pseudo-labels. Our method is independent of the internal structure of HOI detectors, so it can be combined with off-the-shelf methods by training merely 10 additional epochs. With the assistance of our method, multiple methods obtain significant improvements, and new state-of-the-art results are achieved on two benchmarks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 4 Aug 2023.

Builds on

39 of 52 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  7. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  8. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  9. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  10. 2017Learning to Detect Human-Object Interactionscited by 238
  11. 2022Towards Robust Blind Face Restoration with Codebook Lookup Transformercited by 230
  12. 2015Visual Semantic Role Labelingcited by 188
27 more
  1. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  2. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  3. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  4. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  5. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  6. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  7. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  8. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  9. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  10. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  11. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  12. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  13. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  14. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  15. 2017Learning Feature Pyramids for Human Pose Estimationcited by 57
  16. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  17. 2021ISTR: End-to-End Instance Segmentation with Transformerscited by 52
  18. 2022Interactiveness Field in Human-Object Interactionscited by 40
  19. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  20. 2023You Only Segment Once: Towards Real-Time Panoptic Segmentationcited by 40
  21. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  22. 2022Omni-DETR: Omni-Supervised Object Detection with Transformerscited by 28
  23. 2022Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semanticscited by 22
  24. 2022Consistency Learning via Decoding Path Augmentation for Transformers in Human Object Interaction Detectioncited by 16
  25. 2022Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detectioncited by 12
  26. 2022Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detectioncited by 9
  27. 2022Distance Matters in Human-Object Interaction Detectioncited by 8

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detectioncited by 11
  2. 2024A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gapcited by 6
  3. 2025Learning Human-Object Interaction as Groupscited by 5
  4. 2025B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testingcited by 0