mlsanityData to 7 Oct 2026

Agglomerative Transformer for Human-Object Interaction Detection

arXiv 2308.08370 ↗First posted 16 Aug 2023cs.CV cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose an agglomerative Transformer (AGER) that enables Transformer-based human-object interaction (HOI) detectors to flexibly exploit extra instance-level cues in a single-stage and end-to-end manner for the first time. AGER acquires instance tokens by dynamically clustering patch tokens and aligning cluster centers to instances with textual guidance, thus enjoying two benefits: 1) Integrality: each instance token is encouraged to contain all discriminative feature regions of an instance, which demonstrates a significant improvement in the extraction of different instance-level cues and subsequently leads to a new state-of-the-art performance of HOI detection with 36.75 mAP on HICO-Det. 2) Efficiency: the dynamical clustering mechanism allows AGER to generate instance tokens jointly with the feature learning of the Transformer encoder, eliminating the need of an additional object detector or instance decoder in prior methods, thus allowing the extraction of desirable extra cues for HOI detection in a single-stage and end-to-end pipeline. Concretely, AGER reduces GFLOPs by 8.5% and improves FPS by 36%, even compared to a vanilla DETR-like pipeline without extra cue extraction.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 56 of its 70 references that are in the map. First posted on arXiv 16 Aug 2023.

Builds on

56 of 70 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  11. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  12. 2017Focal Loss for Dense Object Detectioncited by 749
38 more
  1. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  2. 2017Deep Layer Aggregationcited by 436
  3. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  4. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  5. 2017Learning to Detect Human-Object Interactionscited by 238
  6. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  7. 2015Visual Semantic Role Labelingcited by 188
  8. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  9. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  10. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  11. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  12. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  13. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  14. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  15. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 126
  16. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 119
  17. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  18. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  19. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  20. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  21. 2021Pose Recognition with Cascade Transformerscited by 99
  22. 2021Reformulating HOI Detection as Adaptive Set Predictioncited by 95
  23. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  24. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  25. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  26. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 79
  27. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  28. 2021Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detectioncited by 58
  29. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  30. 2020Contextual Heterogeneous Graph Network for Human-Object Interaction Detectioncited by 50
  31. 2022End-to-End Human-Gaze-Target Detection with Transformerscited by 47
  32. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
  33. 2020Cascaded Human-Object Interaction Recognitioncited by 40
  34. 2022Interactiveness Field in Human-Object Interactionscited by 40
  35. 2022Human-Object Interaction Detection via Disentangled Transformercited by 40
  36. 2022What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactionscited by 35
  37. 2022MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detectioncited by 34
  38. 2021Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queriescited by 31

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 5.

8 papers.

  1. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
  2. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  3. 2023Joint Gaze-Location and Gaze-Object Detectioncited by 6
  4. 2025Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videoscited by 3
  5. 2025Egocentric Human-Object Interaction Detection: A New Benchmark and Methodcited by 3
  6. 2025RoHOI: Robustness Benchmark for Human-Object Interaction Detectioncited by 3
  7. 2025Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detectioncited by 3
  8. 2025Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMscited by 0