mlsanityData to 7 Oct 2026

Inference-Time Structural Reasoning for Compositional Vision-Language Understanding

arXiv 2603.27349 ↗First posted 28 Mar 2026cs.CV cs.CLcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) excel at image-text retrieval yet persistently fail at compositional reasoning, distinguishing captions that share the same words but differ in relational structure. We present, a unified evaluation and augmentation framework benchmarking four architecturally diverse VLMs,CLIP, BLIP, LLaVA, and Qwen3-VL-8B-Thinking,on the Winoground benchmark under plain and scene-graph-augmented regimes. We introduce a dependency-based TextSceneGraphParser (spaCy) extracting subject-relation-object triples, and a Graph Asymmetry Scorer using optimal bipartite matching to inject structural relational priors. Caption ablation experiments (subject-object masking and swapping) reveal that Qwen3-VL-8B-Thinking achieves a group score of 62.75, far above all encoder-based models, while a proposed multi-turn SG filtering strategy further lifts it to 66.0, surpassing prior open-source state-of-the-art. We analyze the capability augmentation tradeoff and find that SG augmentation benefits already capable models while providing negligible or negative gains for weaker baselines. Code: https://github.com/amartyacodes/Inference-Time-Structural-Reasoning-for-Compositional-Vision-Language-Understanding

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 20 references that are in the map. First posted on arXiv 28 Mar 2026.

Builds on

16 of 20 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models9 Oct
  2. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality7 Apr

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  6. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  7. 2018Image Generation from Scene Graphscited by 324
  8. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  9. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  10. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  11. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  12. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
1 more
  1. 2025SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Datacited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Adding simple structure at inference improves Vision-Language Compositionality
  2. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
  3. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  4. 2024Natural Language Inference Improves Compositionality in Vision-Language Models
  5. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
  6. 2026CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
  7. 20233VL: Using Trees to Improve Vision-Language Models' Interpretability
  8. 2023Text encoders bottleneck compositionality in contrastive vision-language models