mlsanityData to 7 Oct 2026

Exploring Predicate Visual Context in Detecting Human-Object Interactions

arXiv 2308.06202 ↗First posted 11 Aug 2023cs.CV cs.AI cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

Recently, the DETR framework has emerged as the dominant approach for human--object interaction (HOI) research. In particular, two-stage transformer-based HOI detectors are amongst the most performant and training-efficient approaches. However, these often condition HOI classification on object features that lack fine-grained contextual information, eschewing pose and orientation information in favour of visual cues about object identity and box extremities. This naturally hinders the recognition of complex or ambiguous interactions. In this work, we study these issues through visualisations and carefully designed experiments. Accordingly, we investigate how best to re-introduce image features via cross-attention. With an improved query design, extensive exploration of keys and values, and box pair positional embeddings as spatial guidance, our model with enhanced predicate visual context (PViC) outperforms state-of-the-art methods on the HICO-DET and V-COCO benchmarks, while maintaining low training cost.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

42 of 49 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Layer Normalizationcited by 5,579
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2017Focal Loss for Dense Object Detectioncited by 749
  11. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  12. 2021Conditional DETR for Fast Training Convergencecited by 422
30 more
  1. 2017Learning to Detect Human-Object Interactionscited by 238
  2. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  3. 2015Visual Semantic Role Labelingcited by 188
  4. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  5. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  6. 2022DETRs with Hybrid Matchingcited by 149
  7. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  8. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  9. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  10. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  11. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  12. 2016Fully Convolutional Instance-aware Semantic Segmentationcited by 122
  13. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  14. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  15. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  16. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  17. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  18. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  19. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  20. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  21. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  22. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  23. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  24. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  25. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
  26. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  27. 2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34
  28. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  29. 2022Iwin: Human-Object Interaction Detection via Transformer with Irregular Windowscited by 19
  30. 2022Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detectioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 16; 2025: 24; 2026: 14.

56 papers.

Sort:

By area:

  1. 2024Knowledge Composition using Task Vectors with Learned Anisotropic Scalingcited by 37
  2. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  3. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  4. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  5. 2024Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detectioncited by 26
  6. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  7. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  8. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  9. 2025HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptationcited by 12
  10. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 12
  11. 2024Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detectioncited by 11
  12. 2025Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibrationcited by 7
38 more
  1. 2024CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generationcited by 5
  2. 2024A Review of Human-Object Interaction Detectioncited by 5
  3. 2025An Image-like Diffusion Method for Human-Object Interaction Detectioncited by 5
  4. 2025Learning Human-Object Interaction as Groupscited by 5
  5. 2024Temporally Grounding Instructional Diagrams in Unconstrained Videoscited by 4
  6. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Modelscited by 3
  7. 2025SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detectioncited by 3
  8. 2025PhysLab: A Benchmark Dataset for Multi-Granularity Visual Parsing of Physics Experimentscited by 3
  9. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  10. 2025RoHOI: Robustness Benchmark for Human-Object Interaction Detectioncited by 3
  11. 2025No More Sibling Rivalry: Debiasing Human-Object Interaction Detectioncited by 3
  12. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  13. 2024Towards Flexible Visual Relationship Segmentationcited by 2
  14. 2024Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detectioncited by 2
  15. 2024CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Modelscited by 1
  16. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysiscited by 1
  17. 2025Dynamic Scene Understanding from Vision-Language Representationscited by 1
  18. 2025Reference-free Human-Object Interaction Editingcited by 1
  19. 2025DQEN: Dual Query Enhancement Network for DETR-based HOI Detectioncited by 1
  20. 2025Trainee Action Recognition through Interaction Analysis in CCATT Mixed-Reality Trainingcited by 1
  21. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  22. 2026Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognitioncited by 1
  23. 2026Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detectioncited by 1
  24. 2026Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wildcited by 1
  25. 2023DRUformer: Enhancing the driving scene Important object detection with driving relationship self-understandingcited by 0
  26. 2024FreeA: Human-object Interaction Detection using Free Annotation Labelscited by 0
  27. 2024Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Modelcited by 0
  28. 2025B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testingcited by 0
  29. 2025Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integrationcited by 0
  30. 2025On the Robustness of Human-Object Interaction Detection against Distribution Shiftcited by 0
  31. 2025Multimodal Graph Network Modeling for Human-Object Interaction Detection with PDE Graph Diffusioncited by 0
  32. 2025Contextualized Representation Learning for Effective Human-Object Interaction Detectioncited by 0
  33. 2025Incremental Human-Object Interaction Detection with Invariant Relation Representation Learningcited by 0
  34. 2025Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMscited by 0
  35. 2026RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detectioncited by 0
  36. 2026A Study of Failure Modes in Two-Stage Human-Object Interaction Detectioncited by 0
  37. 2026OneHOI: Unifying Human-Object Interaction Generation and Editingcited by 0
  38. 2026Towards Unconstrained Human-Object Interactioncited by 0

50 most cited of 56 All 56 (JSON)