mlsanityData to 14 Oct 2026

PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs

arXiv 2605.23883 ↗First posted 22 May 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite remarkable progress in Multimodal Large Language Models (MLLMs), these models still struggle with fine-grained understanding tasks. In this work, we propose Procedurally Generated Tasks (PGT), a simple data-driven framework that serves a dual purpose: inducing fine-grained visual understanding and acting as a low-cost diagnostic tool to identify the source of perception failures. By overlaying unambiguous geometric primitives on images, PGT generate additional dense supervision that disentangles visual grounding capability from semantic priors. Extensive experiments on relational, quantitative, and 3D/depth understanding benchmarks show that PGT yields remarkable gains across diverse architectures. Instruction tuning MLLMs on LLaVA-v1.5-Instruct augmented with PGT data results in improvements of up to +20% on the What'sUp benchmark and +13.3% on CV-Bench-2D, while maintaining general perception capabilities. Moreover, finetuning state-of-the-art MLLMs on PGT data leads to boosts of up to +5.5% on What'sUp and +8.3% on CV-Bench-2D. These findings demonstrate that PGT effectively address the bottleneck of fine-grained perception, revealing that many spatial reasoning deficits stem from inadequate supervision signals rather than inherent architectural or resolution limitations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 42 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

29 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions18 Jul
  2. 2024Locality Alignment Improves Vision-Language Models14 Oct
  3. 2024Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models22 May
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  5. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Models22 Dec

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  5. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  8. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  9. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  10. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  11. 2018TallyQA: Answering Complex Counting Questionscited by 153
  12. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
12 more
  1. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 90
  2. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
  3. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 66
  4. 2019AI2D-RST: A multimodal corpus of 1000 primary school science diagramscited by 60
  5. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 57
  6. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  7. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  8. 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Mapscited by 23
  9. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 16
  10. 2025Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMscited by 15
  11. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  12. 2025Vision Transformers with Self-Distilled Registerscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model
  2. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  3. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
  4. 2024Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models
  5. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
  6. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model
  7. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  8. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation