mlsanityData to 14 Oct 2026

CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test

arXiv 2610.18569 ↗First posted 13 Oct 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision language models (VLMs) are used in creative tasks such as visual storytelling, design ideation, art interpretation, and advertising. Yet we lack principled ways to measure the lateral reasoning of these models compared to human potential. We introduce CREATE, a semi-automatically constructed benchmark containing 155 visual Remote Associates Test problems. These problems require a solver to find an underlying connection among three seemingly dissimilar images. We conduct a human study (N=25) to validate and ground the quality of our benchmark, and evaluate 11 VLMs. Our findings show that most VLMs underperform humans on this task. Our experiments indicate that the associative ability of VLMs depends on the cue concepts provided in the text and does not universally extend to visual puzzles.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 33 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

9 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns20 Mar

Other references · 8

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  3. 2026DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compressioncited by 76
  4. 2024Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoningcited by 25
  5. 2018Quantifying the visual concreteness of words and topics in multimodal datasetscited by 17
  6. 2018Predicting Concreteness and Imageability of Words Within and Across Languages via Word Embeddingscited by 11
  7. 2023Composition and Deformance: Measuring Imageability with a Text-to-Image Modelcited by 5
  8. 2026"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs
  2. 2026How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning
  3. 2025Probing and Inducing Combinational Creativity in Vision-Language Models
  4. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM
  5. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
  6. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
  7. 2025GuessBench: Sensemaking Multimodal Creativity in the Wild
  8. 2025VLM-Guided Adaptive Negative Prompting for Creative Generation