mlsanityData to 14 Oct 2026

VisAnalog: A Diagnostic Suite for Visual Concept Transfer on Natural Images

arXiv 2605.23141 ↗First posted 22 May 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

A useful test of visual concept learning is not just whether a model can recognize a concept in a single image, but whether it can preserve and manipulate concept-level properties under transformation and transfer them to new scenes. We introduce VisAnalog, a controlled suite for this setting on natural images. Each example instantiates A!:!B::C!:,?: images B and a hidden target image D are produced by applying the same deterministic transformation sequence to source images A and C. Given A, B, and C, a model must answer a multiple-choice question about D. The benchmark contains 617 human-validated questions spanning one- to four-step transformations such as zoom, quadrant swap, rotation, flip, and hue rotation. Across strong proprietary and open-source VLMs, end-to-end accuracy is substantially lower than oracle accuracy when D is directly shown, and degrades sharply as transformation depth increases, while human performance remains near the ceiling. A program-conditioned evaluation further separates failures of relation inference from failures of transformation application, showing that inferring the visual relation from A rightarrow B is the dominant bottleneck, with additional application errors emerging on harder multi-step cases. The dataset is publicly available at https://huggingface.co/datasets/zli99/VisAnalog.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

22 of 40 references.

References · 22

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Segment Anythingcited by 8,234
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  6. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  8. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  11. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  12. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
10 more
  1. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  2. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  3. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 101
  4. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 55
  5. 2023In-Context Analogical Reasoning with Pre-Trained Language Modelscited by 22
  6. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogiescited by 13
  7. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generationcited by 13
  8. 2025BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perceptioncited by 9
  9. 2025Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?cited by 5
  10. 2025VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Show Me Examples: Inferring Visual Concepts from Image Sets
  2. 2026Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap
  3. 2026Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning
  4. 2025Bridging the gap to real-world language-grounded visual concept learning
  5. 2023Concept Decomposition for Visual Exploration and Inspiration
  6. 2024Understanding Visual Concepts Across Models
  7. 2023Cross-Modal Concept Learning and Inference for Vision-Language Models
  8. 2023ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models