# Can We Talk Models Into Seeing the World Differently?

arXiv: https://arxiv.org/abs/2403.09193 · HTML: https://mlsanity.com/paper/2403.09193/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 48 of its 71 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023)

## Built on by (most cited first, 23 of 23)

- [Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift](https://mlsanity.com/paper/2406.18844/) (2024)
- [MMPB: It's Time for Multi-Modal Personalization](https://mlsanity.com/paper/2509.22820/) (2025)
- [ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression](https://mlsanity.com/paper/2509.20234/) (2025)
- [Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It](https://mlsanity.com/paper/2507.13328/) (2025)
- [The in-context inductive biases of vision-language models differ across modalities](https://mlsanity.com/paper/2502.01530/) (2025)
- [Adopting a human developmental visual diet yields robust, shape-based AI vision](https://mlsanity.com/paper/2507.03168/) (2025)
- [Vision-Language Models display a strong gender bias](https://mlsanity.com/paper/2508.11262/) (2025)
- [Thousand-Brains Systems: Sensorimotor Intelligence for Rapid, Robust Learning and Inference](https://mlsanity.com/paper/2507.04494/) (2025)
- [VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes](https://mlsanity.com/paper/2509.25339/) (2025)
- [Same Answer, Different Representations: Hidden instability in VLMs](https://mlsanity.com/paper/2602.06652/) (2026)
- [How Do Training Methods Influence the Utilization of Vision Models?](https://mlsanity.com/paper/2410.14470/) (2024)
- [Shape Bias and Robustness Evaluation via Cue Decomposition for Image Classification and Segmentation](https://mlsanity.com/paper/2503.12453/) (2025)
- [Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech](https://mlsanity.com/paper/2606.01479/) (2026)
- [Automatic benchmarking of large multimodal models via iterative experiment programming](https://mlsanity.com/paper/2406.12321/) (2024)
- [Do large language vision models understand 3D shapes?](https://mlsanity.com/paper/2412.10908/) (2024)
- [Decomposing and Revising What Language Models Generate](https://mlsanity.com/paper/2509.00765/) (2025)
- [Visual Persuasion: What Influences Decisions of Vision-Language Models?](https://mlsanity.com/paper/2602.15278/) (2026)
- [TraversalBench: Challenging Paths to Follow for Vision Language Models](https://mlsanity.com/paper/2604.10999/) (2026)
- [On the Influence of Shape, Texture and Color for Learning Semantic Segmentation](https://mlsanity.com/paper/2410.14878/) (2024)
- [Contrast All the Time: Learning Time Series Representation from Temporal Consistency](https://mlsanity.com/paper/2410.15416/) (2024)
- [TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models](https://mlsanity.com/paper/2511.11831/) (2025)
- [Would you still call this Dax? Novel Visual References in VLMs and Humans](https://mlsanity.com/paper/2606.05409/) (2026)
- [(How) Do MLLMs Report Bistable Images Like Humans?](https://mlsanity.com/paper/2609.13254/) (2026)

## Abstract (arXiv)

Unlike traditional vision-only models, vision language models (VLMs) offer an intuitive way to access visual content through language prompting by combining a large language model (LLM) with a vision encoder. However, both the LLM and the vision encoder come with their own set of biases, cue preferences, and shortcuts, which have been rigorously studied in uni-modal models. A timely question is how such (potentially misaligned) biases and cue preferences behave under multi-modal fusion in VLMs. As a first step towards a better understanding, we investigate a particularly well-studied vision-only bias - the texture vs. shape bias and the dominance of local over global information. As expected, we find that VLMs inherit this bias to some extent from their vision encoders. Surprisingly, the multi-modality alone proves to have important effects on the model behavior, i.e., the joint training and the language querying change the way visual cues are processed. While this direct impact of language-informed training on a model's visual perception is intriguing, it raises further questions on our ability to actively steer a model's output so that its prediction is based on particular visual cues of the user's choice. Interestingly, VLMs have an inherent tendency to recognize objects based on shape information, which is different from what a plain vision encoder would do. Further active steering towards shape-based classifications through language prompts is however limited. In contrast, active VLM steering towards texture-based decisions through simple natural language prompts is often more successful. URL: https://github.com/paulgavrikov/vlm_shapebias