mlsanityData to 14 Oct 2026

Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models

arXiv 2402.12058 ↗First posted 19 Feb 2024cs.CV cs.CLcited by 57

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

State-of-the-art Large Multi-Modal Models (LMMs) have demonstrated exceptional capabilities in vision-language tasks. Despite their advanced functionalities, the performances of LMMs are still limited in challenging scenarios that require complex reasoning with multiple levels of visual information. Existing prompting techniques for LMMs focus on either improving textual reasoning or leveraging tools for image preprocessing, lacking a simple and general visual prompting scheme to promote vision-language coordination in LMMs. In this work, we propose Scaffold prompting that scaffolds coordinates to promote vision-language coordination. Specifically, Scaffold overlays a dot matrix within the image as visual information anchors and leverages multi-dimensional coordinates as textual positional references. Extensive experiments on a wide range of challenging vision-language tasks demonstrate the superiority of Scaffold over GPT-4V with the textual CoT prompting. Our code is released in https://github.com/leixy20/Scaffold.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 44 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

31 of 44 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Segment Anythingcited by 8,234
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  9. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  10. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  11. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  12. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
14 more
  1. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  2. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  4. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  5. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 80
  6. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 80
  7. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 77
  8. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 51
  9. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  10. 2023Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Leadcited by 40
  11. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4Vcited by 38
  12. 2023The Role of Chain-of-Thought in Complex Vision-Language Reasoning Taskcited by 32
  13. 2023An Early Evaluation of GPT-4V(ision)cited by 31
  14. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 26; 2026: 12.

57 papers.

Sort:

By area:

  1. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 206
  2. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 165
  3. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 129
  4. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  5. 2025Visual Planning: Let's Think Only with Imagescited by 85
  6. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 65
  7. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  8. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  9. 2024Interleaved-Modal Chain-of-Thoughtcited by 51
  10. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Modelscited by 50
  11. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 46
  12. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 46
38 more
  1. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 43
  2. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 42
  3. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 38
  4. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 37
  5. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cuescited by 33
  6. 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Modelscited by 31
  7. 2025Multi-Step Visual Reasoning with Visual Tokens Scaling and Verificationcited by 21
  8. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  9. 2024Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Groundingcited by 18
  10. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  11. 2025Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarioscited by 16
  12. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  13. 2024TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigationcited by 15
  14. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generationcited by 13
  15. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  16. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memorycited by 9
  17. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  18. 2025LayerPeeler: Autoregressive Peeling for Layer-wise Image Vectorizationcited by 8
  19. 2025FlySearch: Exploring how vision-language models explorecited by 7
  20. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4ocited by 6
  21. 2025ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoningcited by 6
  22. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoningcited by 5
  23. 2024Layout Generation Agents with Large Language Modelscited by 4
  24. 2024MLLM-Search: A Zero-Shot Approach to Finding People using Multimodal Large Language Modelscited by 4
  25. 2024Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Controlcited by 4
  26. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Modelscited by 3
  27. 2026AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysiscited by 3
  28. 2026Visual Enhanced Depth Scaling for Multimodal Latent Reasoningcited by 3
  29. 2025How Auxiliary Reasoning Unleashes GUI Grounding in VLMscited by 2
  30. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 2
  31. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  32. 2025MobQA: A Benchmark Dataset for Semantic Understanding of Human Mobility Data through Question Answeringcited by 1
  33. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Modelscited by 1
  34. 2026RAWDet-7: A Multi-Scenario Benchmark for Object Detection and Description on Quantized RAW Imagescited by 1
  35. 2026HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Modelscited by 1
  36. 2026Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoningcited by 1
  37. 2026ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Groundingcited by 1
  38. 2025QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understandingcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models
  2. 2026Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
  3. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  4. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
  5. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision
  6. 2023Large Language Models are Visual Reasoning Coordinators
  7. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
  8. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models