mlsanityData to 7 Oct 2026

Visual Semantic Role Labeling

arXiv 1505.04474 ↗First posted 17 May 2015cs.CVcited by 188

Not on the map. Why?

Abstract (arXiv)

In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 May 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 28; 2022: 38; 2023: 47; 2024: 30; 2025: 29; 2026: 15.

188 papers.

Sort:

By area:

  1. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  2. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  3. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  4. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  5. 2022Panoptic Scene Graph Generationcited by 114
  6. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  7. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  8. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  9. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  10. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  11. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  12. 2023OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mappingcited by 83
38 more
  1. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  2. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  3. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  4. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  5. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  6. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  7. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  8. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  9. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  10. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
  11. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  12. 2021Human-like Controllable Image Captioning with Verb-specific Semantic Rolescited by 41
  13. 2023Detecting Human-Object Contact in Imagescited by 41
  14. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  15. 2022Interactiveness Field in Human-Object Interactionscited by 40
  16. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  17. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  18. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  19. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  20. 2023Relational Context Learning for Human-Object Interaction Detectioncited by 35
  21. 2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34
  22. 2022Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactionscited by 34
  23. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  24. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  25. 2023UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detectioncited by 31
  26. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  27. 2023FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detectioncited by 30
  28. 2022HAKE: A Knowledge Engine Foundation for Human Activity Understandingcited by 28
  29. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  30. 2023Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Modelscited by 27
  31. 2024Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detectioncited by 26
  32. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  33. 2025A Survey on Human Interaction Motion Generationcited by 25
  34. 2021Exploiting Scene Graphs for Human-Object Interaction Detectioncited by 22
  35. 2022Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semanticscited by 22
  36. 2022Category-Aware Transformer Network for Better Human-Object Interaction Detectioncited by 22
  37. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
  38. 2023CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detectioncited by 22

50 most cited of 188 All 188 (JSON)