GlyphPattern: An Abstract Pattern Recognition Benchmark for Vision-Language Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Vision-Language Models (VLMs) building upon the foundation of powerful large language models have made rapid progress in reasoning across visual and textual data. While VLMs perform well on vision tasks that they are trained on, our results highlight key challenges in abstract pattern recognition. We present GlyphPattern, a 954 item dataset that pairs 318 human-written descriptions of visual patterns from 40 writing systems with three visual presentation styles. GlyphPattern evaluates abstract pattern recognition in VLMs, requiring models to understand and judge natural language descriptions of visual patterns. GlyphPattern patterns are drawn from a large-scale cognitive science investigation of human writing systems; as a result, they are rich in spatial reference and compositionality. Our experiments show that GlyphPattern is challenging for state-of-the-art VLMs (GPT-4o achieves only 55% accuracy), with marginal gains from few-shot prompting. Our detailed error analysis reveals challenges at multiple levels, including visual processing, natural language understanding, and pattern generalization.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 34 references that are in the map. First posted on arXiv 12 Aug 2024.
Builds on
18 of 34 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2023GPT-4 Technical Reportcited by 18,396
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
- 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
- 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
- 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
- 2018Measuring abstract reasoning in neural networkscited by 166
- 2020Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoningcited by 63
- 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
5 more
- 2022A Benchmark for Compositional Visual Reasoningcited by 43
- 2022Abstract Visual Reasoning with Tangram Shapescited by 43
- 2022Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactionscited by 34
- 2019V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matricescited by 24
- 2023Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real Worldcited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing
- 2025Glyph: Scaling Context Windows via Visual-Text Compression
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?
- 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns
- 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
- 2023An Examination of the Compositionality of Large Generative Vision-Language Models
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
- 2025Visible Yet Unreadable: A Systematic Blind Spot of Vision Language Models Across Writing Systems