mlsanityData to 14 Oct 2026

Abstract Visual Reasoning with Tangram Shapes

arXiv 2211.16492 ↗First posted 29 Nov 2022cs.CL cs.AI cs.CVcited by 43

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k distinct stimuli, is orders of magnitude larger and more diverse than prior resources. It is both visually and linguistically richer, moving beyond whole shape descriptions to include segmentation maps and part labels. We use this resource to evaluate the abstract visual reasoning capacities of recent multi-modal models. We observe that pre-trained weights demonstrate limited abstract reasoning, which dramatically improves with fine-tuning. We also observe that explicitly describing parts aids abstract reasoning for both humans and models, especially when jointly encoding the linguistic and visual inputs. KiloGram is available at https://lil.nlp.cornell.edu/kilogram .

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 58 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

17 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality7 Apr

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
4 more
  1. 2019Characterizing the dynamics of learning in repeated reference gamescited by 34
  2. 2021Learning to communicate about shared procedural abstractionscited by 15
  3. 2022Identifying concept libraries from language about object structurecited by 8
  4. 2021Shades of confusion: Lexical uncertainty modulates ad hoc coordination in an interactive communication taskcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 12; 2025: 10; 2026: 5.

43 papers.

Sort:

By area:

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
  3. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  4. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  5. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  6. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 30
  7. 2023IRFL: Image Recognition of Figurative Languagecited by 23
  8. 2023In-Context Analogical Reasoning with Pre-Trained Language Modelscited by 22
  9. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  10. 2025Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulationscited by 21
  11. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  12. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
31 more
  1. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  2. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 15
  3. 2022Do language models have coherent mental models of everyday things?cited by 13
  4. 2023Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategycited by 12
  5. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Modelscited by 10
  6. 2025Measuring How (Not Just Whether) VLMs Build Common Groundcited by 8
  7. 2026MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Gamescited by 8
  8. 2024CoGen: Learning from Feedback with Coupled Comprehension and Generationcited by 7
  9. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  10. 2024Retrospective Learning from Interactionscited by 6
  11. 2025Probing and Inducing Combinational Creativity in Vision-Language Modelscited by 6
  12. 2023Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourneycited by 5
  13. 2023Generating Language Corrections for Teaching Physical Control Taskscited by 4
  14. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 4
  15. 2023HeGeL: A Novel Dataset for Geo-Location from Hebrew Textcited by 3
  16. 2024Where Do We Go from Here? Multi-scale Allocentric Relational Inference from Natural Spatial Descriptionscited by 3
  17. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  18. 2026TangramSR: Can Vision-Language Models Reason in Continuous Geometric Space?cited by 3
  19. 2023Semantic uncertainty guides the extension of conventions to new referentscited by 2
  20. 2023A Joint Study of Phrase Grounding and Task Performance in Vision and Language Modelscited by 2
  21. 2023Compositional Learning of Visually-Grounded Concepts Using Reinforcementcited by 2
  22. 2025SceneGram: Conceptualizing and Describing Tangrams in Scene Contextcited by 2
  23. 2024Into the Unknown: Generating Geospatial Descriptions for New Environmentscited by 1
  24. 2025mrCAD: Multimodal Refinement of Computer-aided Designscited by 1
  25. 2023Apollo: Zero-shot MultiModal Reasoning with Multiple Expertscited by 0
  26. 2024GlyphPattern: An Abstract Pattern Recognition Benchmark for Vision-Language Modelscited by 0
  27. 2025Ad hoc conventions generalize to new referentscited by 0
  28. 2025Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructionscited by 0
  29. 2025When Diffusion Breaks Constraints: Sequential Autoregressive Generation with RL and MCTScited by 0
  30. 2026Structure-Guided Diffusion Model for EEG-Based Visual Cognition Reconstructioncited by 0
  31. 2026TRACE: Transparent Retrieval for Abstract Concept Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles
  2. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blind
  3. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns
  4. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
  5. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning
  6. 2025Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
  7. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
  8. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity