# ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition

arXiv: https://arxiv.org/abs/2307.00586 · HTML: https://mlsanity.com/paper/2307.00586/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 2 Jul 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Grounded Situation Recognition](https://mlsanity.com/paper/2003.12058/) (2020)
- [Teaching Structured Vision&Language Concepts to Vision&Language Models](https://mlsanity.com/paper/2211.11733/) (2022)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [Situation Recognition with Graph Neural Networks](https://mlsanity.com/paper/1708.04320/) (2017)
- [Reflective Decoding Network for Image Captioning](https://mlsanity.com/paper/1908.11824/) (2019)
- [Collaborative Transformers for Grounded Situation Recognition](https://mlsanity.com/paper/2203.16518/) (2022)
- [Hierarchical Self-supervised Representation Learning for Movie Understanding](https://mlsanity.com/paper/2204.03101/) (2022)
- [Rethinking the Two-Stage Framework for Grounded Situation Recognition](https://mlsanity.com/paper/2112.05375/) (2021)
- [Grounded Situation Recognition with Transformers](https://mlsanity.com/paper/2111.10135/) (2021)
- [Commonly Uncommon: Semantic Sparsity in Situation Recognition](https://mlsanity.com/paper/1612.00901/) (2016)
- [Recurrent Models for Situation Recognition](https://mlsanity.com/paper/1703.06233/) (2017)
- [Grounded Video Situation Recognition](https://mlsanity.com/paper/2210.10828/) (2022)

## Built on by (most cited first, 6 of 6)

- [Situational Scene Graph for Structured Human-centric Situation Understanding](https://mlsanity.com/paper/2410.22829/) (2024)
- [Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos](https://mlsanity.com/paper/2407.20642/) (2024)
- [Dynamic Scene Understanding from Vision-Language Representations](https://mlsanity.com/paper/2501.11653/) (2025)
- [From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition](https://mlsanity.com/paper/2507.14686/) (2025)
- [Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering](https://mlsanity.com/paper/2508.04945/) (2025)
- [The Demon is in Ambiguity: Revisiting Situation Recognition with Single Positive Multi-Label Learning](https://mlsanity.com/paper/2508.21816/) (2025)

## Abstract (arXiv)

Situation Recognition is the task of generating a structured summary of what is happening in an image using an activity verb and the semantic roles played by actors and objects. In this task, the same activity verb can describe a diverse set of situations as well as the same actor or object category can play a diverse set of semantic roles depending on the situation depicted in the image. Hence a situation recognition model needs to understand the context of the image and the visual-linguistic meaning of semantic roles. Therefore, we leverage the CLIP foundational model that has learned the context of images via language descriptions. We show that deeper-and-wider multi-layer perceptron (MLP) blocks obtain noteworthy results for the situation recognition task by using CLIP image and text embedding features and it even outperforms the state-of-the-art CoFormer, a Transformer-based model, thanks to the external implicit visual-linguistic knowledge encapsulated by CLIP and the expressive power of modern MLP block designs. Motivated by this, we design a cross-attention-based Transformer using CLIP visual tokens that model the relation between textual roles and visual entities. Our cross-attention-based Transformer known as ClipSitu XTF outperforms existing state-of-the-art by a large margin of 14.1% on semantic role labelling (value) for top-1 accuracy using imSitu dataset. Similarly, our ClipSitu XTF obtains state-of-the-art situation localization performance. We will make the code publicly available.