# The in-context inductive biases of vision-language models differ across modalities

arXiv: https://arxiv.org/abs/2502.01530 · HTML: https://mlsanity.com/paper/2502.01530/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 6 of 6)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [The broader spectrum of in-context learning](https://mlsanity.com/paper/2412.03782/) (2024)
- [Can We Talk Models Into Seeing the World Differently?](https://mlsanity.com/paper/2403.09193/) (2024)
- [Aligning Machine and Human Visual Representations across Abstraction Levels](https://mlsanity.com/paper/2409.06509/) (2024)

## Built on by (most cited first, 5 of 5)

- [Visual serial processing deficits explain divergences in human and VLM reasoning](https://mlsanity.com/paper/2509.25142/) (2025)
- [Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models](https://mlsanity.com/paper/2510.21740/) (2025)
- [Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect](https://mlsanity.com/paper/2507.10013/) (2025)
- [Symphony of Bias: Exploring Gender Associations with Musical Instruments in Multimodal LLMs](https://mlsanity.com/paper/2607.26355/) (2026)
- [In-Context Learning Can Help Vision-Language Models Overcome Training Priors](https://mlsanity.com/paper/2610.12942/) (2026)

## Abstract (arXiv)

Inductive biases are what allow learners to make guesses in the absence of conclusive evidence. These biases have often been studied in cognitive science using concepts or categories -- e.g. by testing how humans generalize a new category from a few examples that leave the category boundary ambiguous. We use these approaches to study generalization in foundation models during in-context learning. Modern foundation models can condition on both vision and text, and differences in how they interpret and learn from these different modalities is an emerging area of study. Here, we study how their generalizations vary by the modality in which stimuli are presented, and the way the stimuli are described in text. We study these biases with three different experimental paradigms, across three different vision-language models. We find that the models generally show some bias towards generalizing according to shape over color. This shape bias tends to be amplified when the examples are presented visually. By contrast, when examples are presented in text, the ordering of adjectives affects generalization. However, the extent of these effects vary across models and paradigms. These results help to reveal how vision-language models represent different types of inputs in context, and may have practical implications for the use of vision-language models.