Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Existing Large Vision-Language Models (LVLMs) excel at matching concepts across multi-modal inputs but struggle with compositional concepts and high-level relationships between entities. This paper introduces Progressive multi-granular Vision-Language alignments (PromViL), a novel framework to enhance LVLMs' ability in performing grounded compositional visual reasoning tasks. Our approach constructs a hierarchical structure of multi-modal alignments, ranging from simple to complex concepts. By progressively aligning textual descriptions with corresponding visual regions, our model learns to leverage contextual information from lower levels to inform higher-level reasoning. To facilitate this learning process, we introduce a data generation process that creates a novel dataset derived from Visual Genome, providing a wide range of nested compositional vision-language pairs. Experimental results demonstrate that our PromViL framework significantly outperforms baselines on various visual grounding and compositional question answering tasks. The code is available at: https://github.com/lqh52/PromViL.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 40 references that are in the map. First posted on arXiv 11 Dec 2024.
Builds on
9 of 40 references.
References · 9
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2024Mixtral of Expertscited by 1,823
- 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
- 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
- 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
- 2021Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answeringcited by 29
- 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
- 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
- 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
- 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
- 2026Vision-aligned Latent Reasoning for Multi-modal Large Language Model
- 2026Faithful Grounded Visual Reasoning via Learned Proxy-Tokens
- 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding
- 2025COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models