mlsanityData to 14 Oct 2026

AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity

arXiv 2509.14171 ↗First posted 17 Sep 2025cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Recent advancements in multimodal large language models (MLLMs) have garnered significant attention, offering a promising pathway toward artificial general intelligence (AGI). Among the essential capabilities required for AGI, creativity has emerged as a critical trait for MLLMs, with association serving as its foundation. Association reflects a model' s ability to think creatively, making it vital to evaluate and understand. While several frameworks have been proposed to assess associative ability, they often overlook the inherent ambiguity in association tasks, which arises from the divergent nature of associations and undermines the reliability of evaluations. To address this issue, we decompose ambiguity into two types-internal ambiguity and external ambiguity-and introduce AssoCiAm, a benchmark designed to evaluate associative ability while circumventing the ambiguity through a hybrid computational method. We then conduct extensive experiments on MLLMs, revealing a strong positive correlation between cognition and association. Additionally, we observe that the presence of ambiguity in the evaluation process causes MLLMs' behavior to become more random-like. Finally, we validate the effectiveness of our method in ensuring more accurate and reliable evaluations. See Project Page for the data and codes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 52 of its 113 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

52 of 113 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models14 Feb
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Segment Anythingcited by 8,234
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
38 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  6. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  7. 2024Yi: Open Foundation Models by 01.AIcited by 719
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  11. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  13. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  14. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  15. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  16. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  17. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  18. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  19. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  20. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  21. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  22. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 61
  23. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  24. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 55
  25. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  26. 2023Probing the Creativity of Large Language Models: Can models produce divergent semantic association?cited by 33
  27. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  28. 2024Divergent Creativity in Humans and Large Language Modelscited by 31
  29. 2023StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understandingcited by 26
  30. 2022AmbiPun: Generating Humorous Puns with Ambiguous Contextcited by 21
  31. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 19
  32. 2023BRAINTEASER: Lateral Thinking Puzzles for Large Language Modelscited by 18
  33. 2024The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Modelscited by 17
  34. 2024MoExtend: Tuning New Experts for Modality and Task Extensioncited by 12
  35. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
  36. 2024Do Large Language Models Solve ARC Visual Analogies Like People Do?cited by 7
  37. 2024Creativity in AI: Progresses and Challengescited by 7
  38. 2024COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models
  2. 2026CREATE: Testing LLMs for Associative Creativity
  3. 2022Modeling Associative Reasoning Processes
  4. 2024The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs
  5. 2024Enhancing Creativity in Large Language Models through Associative Thinking Strategies
  6. 2025Training Emergent Joint Associations: A Reinforcement Learning Approach to Creative Thinking in Language Models
  7. 2026Playing with Words, Improving with Rewards: Training Language Models for Creative Association
  8. 2025CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity