mlsanityData to 7 Oct 2026

Grounded Situation Recognition with Transformers

arXiv 2111.10135 ↗First posted 19 Nov 2021cs.CV cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Grounded Situation Recognition (GSR) is the task that not only classifies a salient action (verb), but also predicts entities (nouns) associated with semantic roles and their locations in the given image. Inspired by the remarkable success of Transformers in vision tasks, we propose a GSR model based on a Transformer encoder-decoder architecture. The attention mechanism of our model enables accurate verb classification by capturing high-level semantic feature of an image effectively, and allows the model to flexibly deal with the complicated and image-dependent relations between entities for improved noun classification and localization. Our model is the first Transformer architecture for GSR, and achieves the state of the art in every evaluation metric on the SWiG benchmark. Our code is available at https://github.com/jhcho99/gsrtr .

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 19 Nov 2021.

Builds on

25 of 35 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2015Gated Graph Sequence Neural Networkscited by 792
  11. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  12. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
13 more
  1. 2020Meta Pseudo Labelscited by 372
  2. 2018Graph R-CNN for Scene Graph Generationcited by 296
  3. 2019Attention on Attention for Image Captioningcited by 289
  4. 2016Image Captioning with Semantic Attentioncited by 260
  5. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  6. 2020Grounded Situation Recognitioncited by 77
  7. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  8. 2021Fully Convolutional Scene Graph Generationcited by 57
  9. 2014Multi-scale Orderless Pooling of Deep Convolutional Activation Featurescited by 38
  10. 2017Situation Recognition with Graph Neural Networkscited by 34
  11. 2016Single Image Action Recognition using Semantic Body Part Actionscited by 14
  12. 2016Commonly Uncommon: Semantic Sparsity in Situation Recognitioncited by 13
  13. 2017Recurrent Models for Situation Recognitioncited by 13

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 2; 2024: 3; 2025: 3.

14 papers.

Sort:

By area:

  1. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
  2. 2022GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinementcited by 20
  3. 2022Collaborative Transformers for Grounded Situation Recognitioncited by 19
  4. 2021Rethinking the Two-Stage Framework for Grounded Situation Recognitioncited by 16
  5. 2022Grounded Video Situation Recognitioncited by 13
  6. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  7. 2022Ambiguous Images With Human Judgments for Robust Visual Event Classificationcited by 12
  8. 2023ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognitioncited by 6
  9. 2025Event Extraction in Large Language Modelcited by 5
  10. 2024Situational Scene Graph for Structured Human-centric Situation Understandingcited by 4
  11. 2024Event-Customized Image Generationcited by 3
  12. 2024Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videoscited by 2
2 more
  1. 2025From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognitioncited by 0
  2. 2025The Demon is in Ambiguity: Revisiting Situation Recognition with Single Positive Multi-Label Learningcited by 0