# What Makes a Maze Look Like a Maze?

arXiv: https://arxiv.org/abs/2409.08202 · HTML: https://mlsanity.com/paper/2409.08202/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models](https://mlsanity.com/paper/2309.04461/) (2023)
- [Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources](https://mlsanity.com/paper/1511.06973/) (2015)
- [What's Left? Concept Grounding with Logic-Enhanced Foundation Models](https://mlsanity.com/paper/2310.16035/) (2023)
- [Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models](https://mlsanity.com/paper/2305.12476/) (2023)
- [Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning](https://mlsanity.com/paper/2309.10814/) (2023)
- [Visual Madlibs: Fill in the blank Image Generation and Question Answering](https://mlsanity.com/paper/1506.00278/) (2015)
- [Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions](https://mlsanity.com/paper/1809.03707/) (2018)
- [Predicting Motivations of Actions by Leveraging Text](https://mlsanity.com/paper/1406.5472/) (2014)
- [A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models](https://mlsanity.com/paper/2402.18409/) (2024)

## Built on by (most cited first, 11 of 11)

- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation](https://mlsanity.com/paper/2511.01163/) (2025)
- [Building Intelligent Agents with Neuro-Symbolic Concepts](https://mlsanity.com/paper/2505.06191/) (2025)
- [MimeQA: Towards Socially-Intelligent Nonverbal Foundation Models](https://mlsanity.com/paper/2502.16671/) (2025)
- [Aligning Generalisation Between Humans and Machines](https://mlsanity.com/paper/2411.15626/) (2024)
- [A Study of Commonsense Reasoning over Visual Object Properties](https://mlsanity.com/paper/2508.10956/) (2025)
- [Generating Fine Details of Entity Interactions](https://mlsanity.com/paper/2504.08714/) (2025)
- [FractalBench: Diagnosing Visual-Mathematical Reasoning Through Recursive Program Synthesis](https://mlsanity.com/paper/2511.06522/) (2025)
- [A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding](https://mlsanity.com/paper/2512.21414/) (2025)
- [Concept Driven Domain Adaptation: Finding an Abstract Needle in a Haystack](https://mlsanity.com/paper/2610.00973/) (2026)

## Abstract (arXiv)

A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across familiar and unfamiliar contexts, and making predictions or reasoning about them. While off-the-shelf vision-language models excel at making literal interpretations of images (e.g., recognizing object categories such as tree branches), they still struggle to make sense of such visual abstractions (e.g., how an arrangement of tree branches may form the walls of a maze). To address this challenge, we introduce Deep Schema Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At the core of DSG are schemas--dependency graph descriptions of abstract concepts that decompose them into more primitive-level symbols. DSG uses large language models to extract schemas, then hierarchically grounds concrete to abstract components of the schema onto images with vision-language models. The grounded schema is used to augment visual abstraction understanding. We systematically evaluate DSG and different methods in reasoning on our new Visual Abstractions Dataset, which consists of diverse, real-world images of abstract concepts and corresponding question-answer pairs labeled by humans. We show that DSG significantly improves the abstract visual reasoning performance of vision-language models, and is a step toward human-aligned understanding of visual abstractions.