# TraversalBench: Challenging Paths to Follow for Vision Language Models

arXiv: https://arxiv.org/abs/2604.10999 · HTML: https://mlsanity.com/paper/2604.10999/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 14 of 14)

- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [TallyQA: Answering Complex Counting Questions](https://mlsanity.com/paper/1810.12440/) (2018)
- [See What You Are Told: Visual Attention Sink in Large Multimodal Models](https://mlsanity.com/paper/2503.03321/) (2025)
- [Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem](https://mlsanity.com/paper/2411.00238/) (2024)
- [VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge](https://mlsanity.com/paper/2504.10342/) (2025)
- [Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach](https://mlsanity.com/paper/2412.18108/) (2024)
- [Can Large Vision Language Models Read Maps Like a Human?](https://mlsanity.com/paper/2503.14607/) (2025)
- [Can We Talk Models Into Seeing the World Differently?](https://mlsanity.com/paper/2403.09193/) (2024)
- [Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.01419/) (2025)
- [Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images](https://mlsanity.com/paper/2502.13928/) (2025)
- [VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs](https://mlsanity.com/paper/2507.13361/) (2025)
- [Visual Graph Question Answering with ASP and LLMs for Language Parsing](https://mlsanity.com/paper/2502.09211/) (2025)
- [Knot So Simple: A Minimalistic Environment for Spatial Reasoning](https://mlsanity.com/paper/2505.18028/) (2025)

## Built on by (most cited first, 1 of 1)

- [VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following](https://mlsanity.com/paper/2605.15672/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) perform strongly on multimodal benchmarks, but their ability to follow complex visual paths remains under-tested. We introduce TraversalBench, a controlled benchmark for exact visual path traversal. Each instance contains a continuous polyline with a unique start marker and labeled vertices; models must recover the ordered sequence encountered from start to finish. The benchmark balances self-intersection count, tortuosity, vertex count, and nearby confounding lines while limiting reliance on OCR, world knowledge, or open-ended planning. We find that self-intersections are the dominant source of difficulty. A first-crossing analysis localizes failures to crossing points: performance is stable before the first crossing, then drops sharply when the model must resolve the correct continuation. Nearby confounders have weaker but compounding effects, and an auxiliary reading-order benchmark reveals a consistent left-to-right bias. Together, these results characterize how VLMs perceive and fail on visual paths. Finally, we position TraversalBench as a new contribution to the growing line of sustained and precise visual grounding benchmarks for VLMs. Code, benchmark data, and rendered examples are available at https://github.com/clarapetrova/traversalbench.