mlsanityData to 13 Oct 2026

Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models

arXiv 2308.13437 ↗First posted 25 Aug 2023cs.CVcited by 59

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, Multimodal Large Language Models (MLLMs) that enable Large Language Models (LLMs) to interpret images through visual instruction tuning have achieved significant success. However, existing visual instruction tuning methods only utilize image-language instruction data to align the language and image modalities, lacking a more fine-grained cross-modal alignment. In this paper, we propose Position-enhanced Visual Instruction Tuning (PVIT), which extends the functionality of MLLMs by integrating an additional region-level vision encoder. This integration promotes a more detailed comprehension of images for the MLLM. In addition, to efficiently achieve a fine-grained alignment between the vision modules and the LLM, we design multiple data generation strategies to construct an image-region-language instruction dataset. Finally, we present both quantitative experiments and qualitative analysis that demonstrate the superiority of the proposed model. Code and data will be released at https://github.com/PVIT-official/PVIT.

Key facts. Cited by 59 papers in the map (data through 13 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 34 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

29 of 34 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  7. 2023Visual Instruction Tuningcited by 7,827
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,299
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
14 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,180
  3. 2021Grounded Language-Image Pre-trainingcited by 1,152
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 815
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 663
  6. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 615
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 515
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  11. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  12. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  13. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  14. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 18; 2024: 26; 2025: 11; 2026: 4.

59 papers.

Sort:

By area:

  1. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 299
  4. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 207
  5. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  6. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  7. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 135
  8. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  9. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  10. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 100
  11. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  12. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
38 more
  1. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  2. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  3. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  4. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  5. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  6. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  7. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  8. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 41
  9. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  10. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 32
  11. 20253D Aware Region Prompted Vision Language Modelcited by 32
  12. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  13. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuningcited by 28
  14. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  15. 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25
  16. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Modelcited by 24
  17. 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
  18. 2024VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Modelcited by 23
  19. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
  20. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  21. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  22. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  23. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  24. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  25. 2023Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generatorcited by 11
  26. 2024SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Modelscited by 10
  27. 2023Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Modelscited by 8
  28. 2024Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiencycited by 7
  29. 2023Large Scale Foundation Models for Intelligent Manufacturing Applications: A Surveycited by 6
  30. 2024V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLMcited by 6
  31. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reportscited by 6
  32. 2024From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuningcited by 5
  33. 2025Text-guided Visual Prompt DINO for Generic Segmentationcited by 5
  34. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Modelcited by 3
  35. 2024DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolutioncited by 3
  36. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Modelscited by 3
  37. 2025InfraGPT Smart Infrastructure: An End-to-End VLM-Based Framework for Detecting and Managing Urban Defectscited by 3
  38. 2025Grounding Everything in Tokens for Multimodal Large Language Modelscited by 2

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  2. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  3. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  4. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?
  5. 2023VIGC: Visual Instruction Generation and Correction
  6. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  7. 2025Learning to Instruct for Visual Instruction Tuning
  8. 2025Visual Representation Alignment for Multimodal Large Language Models