mlsanityData to 7 Oct 2026

Collaborative Transformers for Grounded Situation Recognition

arXiv 2203.16518 ↗First posted 30 Mar 2022cs.CV cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Grounded situation recognition is the task of predicting the main activity, entities playing certain roles within the activity, and bounding-box groundings of the entities in the given image. To effectively deal with this challenging task, we introduce a novel approach where the two processes for activity classification and entity estimation are interactive and complementary. To implement this idea, we propose Collaborative Glance-Gaze TransFormer (CoFormer) that consists of two modules: Glance transformer for activity classification and Gaze transformer for entity estimation. Glance transformer predicts the main activity with the help of Gaze transformer that analyzes entities and their relations, while Gaze transformer estimates the grounded entities by focusing only on the entities relevant to the activity predicted by Glance transformer. Our CoFormer achieves the state of the art in all evaluation metrics on the SWiG dataset. Training code and model weights are available at https://github.com/jhcho99/CoFormer.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

28 of 44 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2015Gated Graph Sequence Neural Networkscited by 792
  11. 2017Focal Loss for Dense Object Detectioncited by 749
  12. 2017Scene Graph Generation by Iterative Message Passingcited by 462
16 more
  1. 2018Graph R-CNN for Scene Graph Generationcited by 296
  2. 2019Attention on Attention for Image Captioningcited by 289
  3. 2016Image Captioning with Semantic Attentioncited by 260
  4. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  5. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  6. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  7. 2020Grounded Situation Recognitioncited by 77
  8. 2020Normalized and Geometry-Aware Self-Attention Network for Image Captioningcited by 63
  9. 2021Paint Transformer: Feed Forward Neural Painting with Stroke Predictioncited by 61
  10. 2021Human-like Controllable Image Captioning with Verb-specific Semantic Rolescited by 41
  11. 2017Situation Recognition with Graph Neural Networkscited by 34
  12. 2021CTRL-C: Camera calibration TRansformer with Line-Classificationcited by 33
  13. 2021Segmentation-grounded Scene Graph Generationcited by 25
  14. 2021Grounded Situation Recognition with Transformerscited by 14
  15. 2016Commonly Uncommon: Semantic Sparsity in Situation Recognitioncited by 13
  16. 2017Recurrent Models for Situation Recognitioncited by 13

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 6; 2024: 2; 2025: 6.

19 papers.

Sort:

By area:

  1. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
  2. 2022GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinementcited by 20
  3. 2023Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labelingcited by 15
  4. 2022Grounded Video Situation Recognitioncited by 13
  5. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  6. 2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representationcited by 13
  7. 2022Ambiguous Images With Human Judgments for Robust Visual Event Classificationcited by 12
  8. 2023Training Multimedia Event Extraction With Generated Images and Captionscited by 11
  9. 2022Video Event Extraction via Tracking Visual States of Argumentscited by 8
  10. 2023ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognitioncited by 6
  11. 2025Event Extraction in Large Language Modelcited by 5
  12. 2024Situational Scene Graph for Structured Human-centric Situation Understandingcited by 4
7 more
  1. 2023Learning Human-Human Interactions in Images from Weak Textual Supervisioncited by 3
  2. 2025Benchmarking and Improving LVLMs on Event Extraction from Multimedia Documentscited by 3
  3. 2024Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videoscited by 2
  4. 2025Dynamic Scene Understanding from Vision-Language Representationscited by 1
  5. 2025Transformer-based Spatial Grounding: A Comprehensive Surveycited by 0
  6. 2025Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clusteringcited by 0
  7. 2025The Demon is in Ambiguity: Revisiting Situation Recognition with Single Positive Multi-Label Learningcited by 0