# VISaGE: Understanding Visual Generics and Exceptions

arXiv: https://arxiv.org/abs/2510.12548 · HTML: https://mlsanity.com/paper/2510.12548/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples](https://mlsanity.com/paper/2410.14669/) (2024)
- [Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images](https://mlsanity.com/paper/2303.07274/) (2023)
- [Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers](https://mlsanity.com/paper/2109.04448/) (2021)
- [Probing Visual Language Priors in VLMs](https://mlsanity.com/paper/2501.00569/) (2024)
- [MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks](https://mlsanity.com/paper/2212.08158/) (2022)
- [IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations](https://mlsanity.com/paper/2404.01266/) (2024)
- [Perceptual Score: What Data Modalities Does Your Model Perceive?](https://mlsanity.com/paper/2110.14375/) (2021)
- [Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?](https://mlsanity.com/paper/2404.18624/) (2024)
- [Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era](https://mlsanity.com/paper/2506.03994/) (2025)

## Built on by (most cited first, 4 of 4)

- [Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking](https://mlsanity.com/paper/2608.01021/) (2026)
- [DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models](https://mlsanity.com/paper/2606.18557/) (2026)
- [Bears, all bears, and some bears. Language Constraints on Language Models' Inductive Inferences](https://mlsanity.com/paper/2601.09852/) (2026)
- [Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models](https://mlsanity.com/paper/2608.25662/) (2026)

## Abstract (arXiv)

While Vision Language Models (VLMs) learn conceptual representations, in the form of generalized knowledge, during training, they are typically used to analyze individual instances. When evaluation instances are atypical, this paradigm results in tension between two priors in the model. The first is a pragmatic prior that the textual and visual input are both relevant, arising from VLM finetuning on congruent inputs; the second is a semantic prior that the conceptual representation is generally true for instances of the category. In order to understand how VLMs trade off these priors, we introduce a new evaluation dataset, VISaGE, consisting of both typical and exceptional images. In carefully balanced experiments, we show that conceptual understanding degrades when the assumption of congruency underlying the pragmatic prior is violated with incongruent images. This effect is stronger than the effect of the semantic prior when querying about individual instances.