mlsanityData to 7 Oct 2026

Grounded Situation Recognition

arXiv 2003.12058 ↗First posted 26 Mar 2020cs.CVcited by 77

Not on the map. Why?

Abstract (arXiv)

We introduce Grounded Situation Recognition (GSR), a task that requires producing structured semantic summaries of images describing: the primary activity, entities engaged in the activity with their roles (e.g. agent, tool), and bounding-box groundings of entities. GSR presents important technical challenges: identifying semantic saliency, categorizing and localizing a large and diverse set of entities, overcoming semantic sparsity, and disambiguating roles. Moreover, unlike in captioning, GSR is straightforward to evaluate. To study this new task we create the Situations With Groundings (SWiG) dataset which adds 278,336 bounding-box groundings to the 11,538 entity classes in the imsitu dataset. We propose a Joint Situation Localizer and find that jointly predicting situations and groundings with end-to-end training handily outperforms independent training on the entire grounding metric suite with relative gains between 8% and 32%. Finally, we show initial findings on three exciting future directions enabled by our models: conditional querying, visual chaining, and grounded semantic aware image retrieval. Code and data available at https://prior.allenai.org/projects/gsr.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 16; 2023: 18; 2024: 19; 2025: 12; 2026: 5.

77 papers.

Sort:

By area:

  1. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  2. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  3. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  4. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  5. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  6. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  7. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  8. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  9. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  10. 2021Human-like Controllable Image Captioning with Verb-specific Semantic Rolescited by 41
  11. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  12. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
38 more
  1. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  2. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  3. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 24
  4. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
  5. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  6. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  7. 2022GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinementcited by 20
  8. 20233DCoMPaT++: An improved Large-scale 3D Vision Dataset for Compositional Recognitioncited by 20
  9. 2024InfiniBench: A Benchmark for Large Multi-Modal Models in Long-Form Movies and TV Showscited by 20
  10. 2021Joint Multimedia Event Extraction from Video and Articlecited by 19
  11. 2022Collaborative Transformers for Grounded Situation Recognitioncited by 19
  12. 2024UMIE: Unified Multimodal Information Extraction with Instruction Tuningcited by 17
  13. 2021Rethinking the Two-Stage Framework for Grounded Situation Recognitioncited by 16
  14. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  15. 2021Grounded Situation Recognition with Transformerscited by 14
  16. 2022Grounded Video Situation Recognitioncited by 13
  17. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  18. 2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representationcited by 13
  19. 2022Ambiguous Images With Human Judgments for Robust Visual Event Classificationcited by 12
  20. 2024INTRA: Interaction Relationship-aware Weakly Supervised Affordance Groundingcited by 12
  21. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  22. 2023Training Multimedia Event Extraction With Generated Images and Captionscited by 11
  23. 2023World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Modelscited by 10
  24. 2024A Survey of Video Datasets for Grounded Event Understandingcited by 9
  25. 2025ChartQA-X: Generating Explanations for Visual Chart Reasoningcited by 9
  26. 2022Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?cited by 8
  27. 2022Video Event Extraction via Tracking Visual States of Argumentscited by 8
  28. 2024Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainercited by 7
  29. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  30. 2023ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognitioncited by 6
  31. 2024MMUTF: Multimodal Multimedia Event Argument Extraction with Unified Template Fillingcited by 6
  32. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  33. 2023Context Does Matter: End-to-end Panoptic Narrative Grounding with Deformable Attention Refined Matching Networkcited by 5
  34. 2023Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Promptingcited by 5
  35. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 5
  36. 2025Event Extraction in Large Language Modelcited by 5
  37. 2023Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categoriescited by 4
  38. 2023Location-Aware Visual Question Generation with Lightweight Modelscited by 4

50 most cited of 77 All 77 (JSON)