mlsanityData to 14 Oct 2026

RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension

arXiv 2308.02299 ↗First posted 3 Aug 2023cs.CVcited by 27

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this work, we investigate extending the comprehension of Multi-modal Large Language Models (MLLMs) to regional objects. To this end, we propose to extract features corresponding to regional objects as soft prompts for LLM, which provides a straightforward and scalable approach and eliminates the need for LLM fine-tuning. To effectively extract regional features from regular image features and irregular point cloud features, we present a novel and unified position-assisted feature extraction module. Furthermore, training an MLLM from scratch is highly time-consuming. Thus, we propose incrementally extending existing pre-trained MLLMs to comprehend more modalities and the regional objects of those modalities. Specifically, we freeze the Q-Former from BLIP-2, an impressive MLLM, and optimize the modality-specific Lora parameters in Q-Former and LLM for each newly introduced modality. The freezing of the Q-Former eliminates the need for extensive pre-training on massive image-text data. The freezed Q-Former pre-trained from massive image-text data is also beneficial for the pre-training on image-region-text data. We name our framework RegionBLIP. We pre-train RegionBLIP on image-region-text, point-cloud-text, and point-cloud-region-text data. Experimental results verify that Ours can preserve the image comprehension capability of BILP-2 and further gain a comprehension of the newly introduced point cloud modality and regional objects. The Data, Code, and Pre-trained models will be available at https://github.com/mightyzau/RegionBLIP.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 43 references that are in the map. First posted on arXiv 3 Aug 2023.

Builds on

36 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  2. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  5. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Segment Anythingcited by 8,234
  5. 2017Mask R-CNNcited by 7,277
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
19 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  4. 2016Modeling Context in Referring Expressionscited by 1,063
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  10. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  11. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  13. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  14. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  15. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  16. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  17. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  18. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  19. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 10; 2025: 7; 2026: 2.

27 papers.

Sort:

By area:

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  3. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  4. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 140
  5. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  6. 2023Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detectioncited by 39
  7. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  8. 20253D Aware Region Prompted Vision Language Modelcited by 33
  9. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 32
  10. 2023Segment and Caption Anythingcited by 30
  11. 2024Unified Language-driven Zero-shot Domain Adaptationcited by 30
  12. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
15 more
  1. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
  2. 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMscited by 24
  3. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  4. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  5. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  6. 2023InfMLLM: A Unified Framework for Visual-Language Taskscited by 11
  7. 2025EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Promptingcited by 11
  8. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  9. 2024G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarioscited by 8
  10. 2025Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encodingcited by 3
  11. 2025Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptionscited by 1
  12. 2025SA-Person: Text-Based Person Retrieval with Scene-aware Re-rankingcited by 1
  13. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  14. 2026Test-Time Computing for Referring Multimodal Large Language Modelscited by 1
  15. 2024VP-MEL: Visual Prompts Guided Multimodal Entity Linkingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
  2. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations
  3. 2024Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion
  4. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
  5. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
  6. 2026Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs