# Situation Recognition with Graph Neural Networks

arXiv: https://arxiv.org/abs/1708.04320 · HTML: https://mlsanity.com/paper/1708.04320/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2017.

## Built on by (most cited first, 30 of 34)

- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement](https://mlsanity.com/paper/2208.08965/) (2022)
- [Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networks](https://mlsanity.com/paper/2107.10234/) (2021)
- [Joint Multimedia Event Extraction from Video and Article](https://mlsanity.com/paper/2109.12776/) (2021)
- [Collaborative Transformers for Grounded Situation Recognition](https://mlsanity.com/paper/2203.16518/) (2022)
- [GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering](https://mlsanity.com/paper/2104.10283/) (2021)
- [Rethinking the Two-Stage Framework for Grounded Situation Recognition](https://mlsanity.com/paper/2112.05375/) (2021)
- [Grounded Situation Recognition with Transformers](https://mlsanity.com/paper/2111.10135/) (2021)
- [GraphFM: Graph Factorization Machines for Feature Interaction Modeling](https://mlsanity.com/paper/2105.11866/) (2021)
- [Grounded Video Situation Recognition](https://mlsanity.com/paper/2210.10828/) (2022)
- [Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments](https://mlsanity.com/paper/2307.07757/) (2023)
- [ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation](https://mlsanity.com/paper/2311.13258/) (2023)
- [DenseGAP: Graph-Structured Dense Correspondence Learning with Anchor Points](https://mlsanity.com/paper/2112.06910/) (2021)
- [Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting](https://mlsanity.com/paper/2101.02385/) (2021)
- [Neural Message Passing for Visual Relationship Detection](https://mlsanity.com/paper/2208.04165/) (2022)
- [ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition](https://mlsanity.com/paper/2307.00586/) (2023)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [Hybrid Reasoning Network for Video-based Commonsense Captioning](https://mlsanity.com/paper/2108.02365/) (2021)
- [Graph Neural Networks in Computer Vision -- Architectures, Datasets and Common Approaches](https://mlsanity.com/paper/2212.10207/) (2022)
- [Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categories](https://mlsanity.com/paper/2308.10562/) (2023)
- [Federated Graph Learning for EV Charging Demand Forecasting with Personalization Against Cyberattacks](https://mlsanity.com/paper/2405.00742/) (2024)
- [Situational Scene Graph for Structured Human-centric Situation Understanding](https://mlsanity.com/paper/2410.22829/) (2024)
- [Beyond Grounding: Extracting Fine-Grained Event Hierarchies Across Modalities](https://mlsanity.com/paper/2206.07207/) (2022)
- [Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos](https://mlsanity.com/paper/2407.20642/) (2024)
- [Learning Complex Users' Preferences for Recommender Systems](https://mlsanity.com/paper/2107.01529/) (2021)
- [Border-SegGCN: Improving Semantic Segmentation by Refining the Border Outline using Graph Convolutional Network](https://mlsanity.com/paper/2109.05353/) (2021)
- [Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision Transformers for High-Level Image Classification](https://mlsanity.com/paper/2402.19339/) (2024)
- [Context-aware short-term interest first model for session-based recommendation](https://mlsanity.com/paper/2103.15514/) (2021)
- [Frame Shift Prediction](https://mlsanity.com/paper/2201.01837/) (2022)

## Abstract (arXiv)

We address the problem of recognizing situations in images. Given an image, the task is to predict the most salient verb (action), and fill its semantic roles such as who is performing the action, what is the source and target of the action, etc. Different verbs have different roles (e.g. attacking has weapon), and each role can take on many possible values (nouns). We propose a model based on Graph Neural Networks that allows us to efficiently capture joint dependencies between roles using neural networks defined on a graph. Experiments with different graph connectivities show that our approach that propagates information between roles significantly outperforms existing work, as well as multiple baselines. We obtain roughly 3-5% improvement over previous work in predicting the full situation. We also provide a thorough qualitative analysis of our model and influence of different roles in the verbs.