mlsanityData to 7 Oct 2026

Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?

arXiv 2409.17080 ↗First posted 25 Sep 2024cs.CV cs.CLcited by 3

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Large vision-language models (VLMs) have become state-of-the-art for many computer vision tasks, with in-context learning (ICL) as a popular adaptation strategy for new ones. But can VLMs learn novel concepts purely from visual demonstrations, or are they limited to adapting to the output format of ICL examples? We propose a new benchmark we call Spatial Visual Ambiguity Tasks (SVAT) that challenges state-of-the-art VLMs to learn new visuospatial tasks in-context. We find that VLMs fail to do this zero-shot, and sometimes continue to fail after finetuning. However, adding simpler data to the training by curriculum learning leads to improved ICL performance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 34 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

31 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  3. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs11 Apr
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  5. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  6. 2022Visual Spatial Reasoning30 Apr

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2023Larger language models do in-context learning differentlycited by 386
  12. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
13 more
  1. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  2. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  3. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  4. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  5. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  6. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  7. 2019Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translationcited by 39
  8. 2021Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generationcited by 35
  9. 2021Curriculum learning for language modelingcited by 32
  10. 2021Curriculum Learning for Vision-and-Language Navigationcited by 18
  11. 2021Weakly Supervised Relative Spatial Reasoning for Visual Question Answeringcited by 12
  12. 2022Curriculum Learning for Data-Efficient Vision-Language Alignmentcited by 11
  13. 2023Improving Vision-and-Language Reasoning via Spatial Relations Modelingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  2. 2026Large Language Model Reasoning Failurescited by 20
  3. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  2. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  3. 2026Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
  4. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?
  5. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  6. 2026WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning
  7. 2025Visual Spatial Tuning
  8. 2024I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction