mlsanityData to 14 Oct 2026

Enhancing Multimodal Large Language Models with Multi-instance Visual Prompt Generator for Visual Representation Enrichment

arXiv 2406.02987 ↗First posted 5 Jun 2024cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved SOTA performance in various visual language tasks by fusing the visual representations with LLMs leveraging some visual adapters. In this paper, we first establish that adapters using query-based Transformers such as Q-former is a simplified Multi-instance Learning method without considering instance heterogeneity/correlation. We then propose a general component termed Multi-instance Visual Prompt Generator (MIVPG) to incorporate enriched visual representations into LLMs by taking advantage of instance correlation between images or patches for the same sample. Quantatitive evaluation on three public vision-language (VL) datasets from different scenarios shows that the proposed MIVPG improves Q-former in main VL tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 49 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

36 of 49 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
24 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2017Deep Setscited by 1,421
  6. 2018Attention-based Deep Multiple Instance Learningcited by 1,095
  7. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
  8. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 555
  9. 2016Self-critical Sequence Training for Image Captioningcited by 547
  10. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 478
  11. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  12. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  13. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  14. 2016Image Captioning with Semantic Attentioncited by 260
  15. 2022DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classificationcited by 256
  16. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  17. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  18. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  19. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  20. 2016Multiple Instance Learning: A Survey of Problem Characteristics and Applicationscited by 164
  21. 2015Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classificationcited by 159
  22. 2016Revisiting Multiple Instance Neural Networkscited by 119
  23. 2016Boosting Image Captioning with Attributescited by 76
  24. 2022Inference of captions from histopathological patchescited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  2. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
  3. 2024Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modalitycited by 2
  4. 2026How to Utilize Complementary Vision-Text Information for 2D Structure Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
  2. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge
  3. 2023MLLMs-Augmented Visual-Language Representation Learning
  4. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  5. 2025Visual Large Language Models for Generalized and Specialized Applications
  6. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  7. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  8. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models