mlsanityData to 7 Oct 2026

Focusing on what to decode and what to train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisor

arXiv 2307.02291 ↗First posted 5 Jul 2023cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Recent transformer-based methods achieve notable gains in the Human-object Interaction Detection (HOID) task by leveraging the detection of DETR and the prior knowledge of Vision-Language Model (VLM). However, these methods suffer from extended training times and complex optimization due to the entanglement of object detection and HOI recognition during the decoding process. Especially, the query embeddings used to predict both labels and boxes suffer from ambiguous representations, and the gap between the prediction of HOI labels and verb labels is not considered. To address these challenges, we introduce SOV-STG-VLA with three key components: Subject-Object-Verb (SOV) decoding, Specific Target Guided (STG) denoising, and a Vision-Language Advisor (VLA). Our SOV decoders disentangle object detection and verb recognition with a novel interaction region representation. The STG denoising strategy learns label embeddings with ground-truth information to guide the training and inference. Our SOV-STG achieves a fast convergence speed and high accuracy and builds a foundation for the VLA to incorporate the prior knowledge of the VLM. We introduce a vision advisor decoder to fuse both the interaction region information and the VLM's vision knowledge and a Verb-HOI prediction bridge to promote interaction representation learning. Our VLA notably improves our SOV-STG and achieves SOTA performance with one-sixth of training epochs compared to recent SOTA. Code and models are available at https://github.com/cjw2021/SOV-STG-VLA

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 55 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

47 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models28 Mar

Other references · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  10. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  11. 2017Learning to Detect Human-Object Interactionscited by 238
  12. 2017Detecting and Recognizing Human-Object Interactionscited by 199
34 more
  1. 2015Visual Semantic Role Labelingcited by 188
  2. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  3. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  4. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  5. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  6. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  7. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  8. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  9. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  10. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  11. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  12. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  13. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  14. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  15. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  16. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  17. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  18. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  19. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  20. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
  21. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  22. 2022Interactiveness Field in Human-Object Interactionscited by 40
  23. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  24. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  25. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 35
  26. 2023FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detectioncited by 30
  27. 2021QAHOI: Query-Based Anchors for Human-Object Interaction Detectioncited by 28
  28. 2020Polysemy Deciphering Network for Robust Human-Object Interaction Detectioncited by 25
  29. 2022Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semanticscited by 22
  30. 2022Category-Aware Transformer Network for Better Human-Object Interaction Detectioncited by 22
  31. 2023Category Query Learning for Human-Object Interaction Classificationcited by 21
  32. 2023Boosting Human-Object Interaction Detection with Text-to-Image Diffusion Modelcited by 21
  33. 2022Towards Hard-Positive Query Mining for DETR-based Human-Object Interaction Detectioncited by 12
  34. 2021The Overlooked Classifier in Human-Object Interaction Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awarenesscited by 7
  2. 2025HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detectioncited by 0
  3. 2026HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detectioncited by 0
  4. 2026HEIR: Learning Human-Entity Interactions with Functional Rolescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priors
  2. 2024Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration
  3. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis
  4. 2025Talk in Pieces, See in Whole: Disentangled and Hierarchical Representation Learning in Language-based Object Detection
  5. 2025Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
  6. 2024CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models
  7. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection
  8. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection