mlsanityData to 7 Oct 2026

KamonBench: A Grammar-Based Dataset for Evaluating Compositional Factor Recovery in Vision-Language Models

arXiv 2605.13322 ↗First posted 13 May 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Kamon (family crests) are an important part of Japanese culture and a natural test case for compositional visual recognition: each crest combines a small number of symbolic choices, but the space of possible descriptions is sparse. We introduce KamonBench, a grammar-based image-to-structure benchmark with 20,000 synthetic composite crests and auxiliary component examples. Each composite crest is paired with a formal kamon description language - "kamon y=ogo" - description, a segmented Japanese analysis, an English translation, and a non-linguistic program code. Because each synthetic crest is generated from known factors, namely container, modifier, and motif, KamonBench supports evaluation beyond caption-level accuracy: direct program-code factor metrics, controlled factor-pair recombination splits, counterfactual motif-sensitivity groups under fixed container-modifier contexts, and linear probes of factor accessibility. We include baseline results for a ViT encoder/Transformer decoder and two VGG n-gram decoders, with and without learned positional masks. KamonBench therefore provides a controlled testbed for sparse compositional visual recognition and factor recovery in vision-language models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 26 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

8 of 26 references.

References · 8

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  6. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  7. 2018Towards a Definition of Disentangled Representationscited by 294
  8. 2019Compositionality decomposed: how do neural networks generalise?cited by 264

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?
  2. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  3. 2024Causal Graphical Models for Vision-Language Compositional Understanding
  4. 2025WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Models
  5. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks
  6. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese
  7. 2026JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
  8. 20233VL: Using Trees to Improve Vision-Language Models' Interpretability