Vision-Language Asymmetry in Bistable Image Captioning
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Wittgenstein's duck-rabbit poses a question for vision-language models: when a model captions an ambiguous image, where in the model is the commitment to one aspect made? We address this with a 3,320-generation behavioral baseline over 83 bistable stimuli that surfaces three regimes (default-dominant, force-dominant, force-balanced) under neutral vs forced-choice prompting, then probe the underlying representations using a TopK sparse autoencoder we train on the CLIP layer that LLaVA-1.6-7B actually consumes (validation EV 0.93). Across 69 bistable stimuli with both per-aspect feature pools available, 72% (50/69) show simultaneous activation of both pools at the vision tower, including 12/12 default-dominant duck/rabbit and 7/8 force-balanced young/old. Causal steering at CLIP layer 22 flips captions on default-dominant stimuli (33% rabbit-flip rate under a fluency guard) but cannot flip captions on force-balanced young/old at any tested coefficient, despite their vision-side superposition. The dominance bottleneck lives downstream of the vision tower; the gap between vision-side representation and language-side commitment is an empirical handle on the seeing/seeing-as distinction. We also flag a methodological note: rank-based statistics on TopK SAE outputs require tie-corrected ranking to avoid silent row-order bias.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 14 references that are in the map. First posted on arXiv 6 Jun 2026.
Builds on
7 of 14 references.
Closest ancestors · 2
Same direction, newest first
- 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models3 Apr
- 2024Evaluating Vision-Language Models on Bistable Images29 May
Other references · 5
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2025Steering CLIP's vision transformer with sparse autoencoderscited by 33
- 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 17
- 2025Mechanistic Interpretability Needs Philosophycited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026(How) Do MLLMs Report Bistable Images Like Humans?
- 2025Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models
- 2026A Framework for Generating Semantically Ambiguous Images to Probe Human and Machine Perception
- 2026Attention Alignment Between Humans and Vision-Language Models
- 2025Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models
- 2026Visual Access Boundaries in Vision-Language Model Reasoning
- 2025Investigating Spatial Attention Bias in Vision-Language Models
- 2026Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits