mlsanityData to 14 Oct 2026

GuessBench: Sensemaking Multimodal Creativity in the Wild

arXiv 2506.00814 ↗First posted 1 Jun 2025cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We propose GuessBench, a novel benchmark that evaluates Vision Language Models (VLMs) on modeling the pervasive, noisy, and pluralistic human creativity. GuessBench sources data from "Guess the Build", an online multiplayer Minecraft minigame where one player constructs a Minecraft build given a concept (e.g. caterpillar) and others try to guess it with natural language hints, presenting a pristine testbed for sensemaking creativity in the wild with VLMs acting as guessers. We curate 1500 images from the actual gameplay and design 2000 problems spanning static and dynamic image settings, natural language hints of varying completeness, and more. Extensive experiments with six open/API VLMs and five reasoning enhancement approaches demonstrate that GuessBench presents a uniquely challenging task in creativity modeling: even the start-of-the-art GPT-4o is incorrect on 34% of instances, while we observe a huge performance gap (13.87% vs. 53.93% on average) between open and API models. When used as a resource to improve VLMs, fine-tuning on the reasoning traces for GuessBench problems improves visual perception tasks by 15.36% on average. Further analysis reveals that VLM performance in creativity sensemaking correlates with the frequency of the concept in training data, while the accuracy drops sharply for concepts in underrepresented cultural contexts and low-resource languages.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 90 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

70 of 90 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Probing and Inducing Combinational Creativity in Vision-Language Models17 Apr
  2. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models25 Jun
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 66

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  8. 2016A Diagram Is Worth A Dozen Imagescited by 804
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  11. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  12. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
38 more
  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  2. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  3. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  4. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  5. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  6. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  7. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  8. 2024An analytic theory of creativity in convolutional diffusion modelscited by 104
  9. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  10. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  11. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 82
  12. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
  13. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  14. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 61
  15. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 59
  16. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  17. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 44
  18. 2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 42
  19. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 40
  20. 2024VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraftcited by 35
  21. 2023MacGyver: Are Large Language Models Creative Problem Solvers?cited by 34
  22. 2023Probing the Creativity of Large Language Models: Can models produce divergent semantic association?cited by 33
  23. 2023HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Modelscited by 29
  24. 2024Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?cited by 27
  25. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 26
  26. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 25
  27. 2025Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policycited by 25
  28. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problemscited by 24
  29. 2022Augmenting Scientific Creativity with an Analogical Search Enginecited by 22
  30. 2024HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playingcited by 22
  31. 2024Collective Critics for Creative Story Generationcited by 22
  32. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoningcited by 21
  33. 2024Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Modelscited by 19
  34. 2023Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Datasetcited by 17
  35. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  36. 2023TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Samplingcited by 16
  37. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraftcited by 14
  38. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMcited by 14

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Do You Get the Hint? Benchmarking LLMs on the Board Game Conceptcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing
  2. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models
  3. 2026CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test
  4. 2026Advancing Creative Physical Intelligence in Large Multimodal Models
  5. 2026How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning
  6. 2026PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models
  7. 2023Creative Agents: Empowering Agents with Imagination for Creative Tasks
  8. 2024APT: Architectural Planning and Text-to-Blueprint Construction Using Large Language Models for Open-World Agents