mlsanityData to 7 Oct 2026

Context-Dependent Affordance Reports in Vision-Language Models

arXiv 2603.04419 ↗First posted 14 Feb 2026cs.CL cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models produce different object and use descriptions under different persona prompts, but low overlap alone does not identify an affordance effect. We audit an earlier seven-prompt study and add matched-question controls. In the historical Qwen pilot, 363 of 3,213 parsed responses contain empty object lists. These affect 2,037 of 9,244 comparisons, with the implementation assigning zero lexical overlap to every affected pair. Conditioning on nonempty reports raises pooled word Jaccard from 0.095 to 0.121 and sentence cosine from 0.415 to 0.511. A previously named chef-specific Tucker factor loses its concentrated loading under missing-cell and complete-nonempty analyses. We withdraw the functional-manifold interpretation and the conversion of similarity scores into percentages of meaning. A new experiment uses 48 images absent from the original pilot, four personas, a shared three-object task, two wordings, and two requested seeds in each of two model configurations. Qwen3.5-9B's persona-minus-wording cosine-distance contrast is 0.0146 (95% CI [-0.0003, 0.0295]; 37 complete images). Ollama llava:13b's persona-minus-wording cosine-distance contrast is -0.0266 (95% CI [-0.0412, -0.0121]; 32 complete images). Persona-associated variation does not uniformly exceed wording or sampling variation. The study provides a reproducible analysis of context-conditioned reports while separating response availability, content similarity, and the limits of inference from linguistic outputs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 14 references that are in the map. First posted on arXiv 14 Feb 2026.

Builds on

7 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Analyzing the Sensitivity of Vision Language Models in Visual Question Answering28 Jul
  2. 2024AffordanceLLM: Grounding Affordance from Vision Language Models12 Jan

Other references · 5

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2018Show, Control and Tell: A Framework for Generating Controllable and Grounded Captionscited by 65
  5. 2025Affordance Benchmark for MLLMscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Text
  2. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
  3. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation
  4. 2026Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
  5. 2026Persona Prompting in Multimodal Urban Perception: Descriptive Convergence and Interpretive Variation
  6. 2025The in-context inductive biases of vision-language models differ across modalities
  7. 2026Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models
  8. 2026I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models