# VASR: Visual Analogies of Situation Recognition

arXiv: https://arxiv.org/abs/2212.04542 · HTML: https://mlsanity.com/paper/2212.04542/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 64 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [Grounded Situation Recognition](https://mlsanity.com/paper/2003.12058/) (2020)
- [Analogies Explained: Towards Understanding Word Embeddings](https://mlsanity.com/paper/1901.09813/) (2019)
- [Do Neural Language Representations Learn Physical Commonsense?](https://mlsanity.com/paper/1908.02899/) (2019)
- [English Machine Reading Comprehension Datasets: A Survey](https://mlsanity.com/paper/2101.10421/) (2021)
- [Solving Bongard Problems with a Visual Language and Pragmatic Reasoning](https://mlsanity.com/paper/1804.04452/) (2018)
- [WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models](https://mlsanity.com/paper/2207.12576/) (2022)
- [VISALOGY: Answering Visual Analogy Questions](https://mlsanity.com/paper/1510.08973/) (2015)

## Built on by (most cited first, 22 of 22)

- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images](https://mlsanity.com/paper/2303.07274/) (2023)
- [StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding](https://mlsanity.com/paper/2310.12874/) (2023)
- [IRFL: Image Recognition of Figurative Language](https://mlsanity.com/paper/2303.15445/) (2023)
- [In-Context Analogical Reasoning with Pre-Trained Language Models](https://mlsanity.com/paper/2305.17626/) (2023)
- [AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies](https://mlsanity.com/paper/2402.12370/) (2024)
- [LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoning](https://mlsanity.com/paper/2502.11176/) (2025)
- [Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models](https://mlsanity.com/paper/2407.19474/) (2024)
- [ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies](https://mlsanity.com/paper/2403.01139/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems](https://mlsanity.com/paper/2508.12026/) (2025)
- [MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval](https://mlsanity.com/paper/2509.26378/) (2025)
- [Benchmarking VLMs' Reasoning About Persuasive Atypical Images](https://mlsanity.com/paper/2409.10719/) (2024)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning](https://mlsanity.com/paper/2503.00043/) (2025)
- [DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities with Diffusion-Generated Synthetic Benchmarks](https://mlsanity.com/paper/2406.04470/) (2024)
- [Any four real numbers are on all fours with analogy](https://mlsanity.com/paper/2407.18770/) (2024)
- [Aligning Generalisation Between Humans and Machines](https://mlsanity.com/paper/2411.15626/) (2024)
- [Beyond Input-Output: Rethinking Creativity through Design-by-Analogy in Human-AI Collaboration](https://mlsanity.com/paper/2602.09423/) (2026)
- [Advancing Generalization Across a Variety of Abstract Visual Reasoning Tasks](https://mlsanity.com/paper/2505.13391/) (2025)
- [Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks](https://mlsanity.com/paper/2606.10967/) (2026)
- [SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interaction](https://mlsanity.com/paper/2608.27461/) (2026)

## Abstract (arXiv)

A core process in human cognition is analogical mapping: the ability to identify a similar relational structure between different situations. We introduce a novel task, Visual Analogies of Situation Recognition, adapting the classical word-analogy task into the visual domain. Given a triplet of images, the task is to select an image candidate B' that completes the analogy (A to A' is like B to what?). Unlike previous work on visual analogy that focused on simple image transformations, we tackle complex analogies requiring understanding of scenes. We leverage situation recognition annotations and the CLIP model to generate a large set of 500k candidate analogies. Crowdsourced annotations for a sample of the data indicate that humans agree with the dataset label ~80% of the time (chance level 25%). Furthermore, we use human annotations to create a gold-standard dataset of 3,820 validated analogies. Our experiments demonstrate that state-of-the-art models do well when distractors are chosen randomly (~86%), but struggle with carefully chosen distractors (~53%, compared to 90% human accuracy). We hope our dataset will encourage the development of new analogy-making models. Website: https://vasr-dataset.github.io/