mlsanityData to 7 Oct 2026

MIVC: Multiple Instance Visual Component for Visual-Language Models

arXiv 2312.17109 ↗First posted 28 Dec 2023cs.CV cs.AI cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models have been widely explored across a wide range of tasks and achieve satisfactory performance. However, it's under-explored how to consolidate entity understanding through a varying number of images and to align it with the pre-trained language models for generative tasks. In this paper, we propose MIVC, a general multiple instance visual component to bridge the gap between various image inputs with off-the-shelf vision-language models by aggregating visual representations in a permutation-invariant fashion through a neural network. We show that MIVC could be plugged into the visual-language models to improve the model performance consistently on visual question answering, classification and captioning tasks on a public available e-commerce dataset with multiple images per product. Furthermore, we show that the component provides insight into the contribution of each image to the downstream tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 40 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

31 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
17 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  4. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  5. 2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
  6. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 473
  7. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  8. 2016Image Captioning with Semantic Attentioncited by 260
  9. 2022DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classificationcited by 254
  10. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  11. 2016Multiple Instance Learning: A Survey of Problem Characteristics and Applicationscited by 163
  12. 2015Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classificationcited by 159
  13. 2016Revisiting Multiple Instance Neural Networkscited by 119
  14. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  15. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  16. 2018Image Captioningcited by 14
  17. 2023Catalog Phrase Grounding (CPG): Grounding of Product Textual Attributes in Product Images for e-commerce Vision-Language Applicationscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Drivingcited by 52

Similar, not cited

Close in wording, no citation either way

  1. 2022cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillation
  2. 2024Enhancing Multimodal Large Language Models with Multi-instance Visual Prompt Generator for Visual Representation Enrichment
  3. 2025Improved Alignment of Modalities in Large Vision Language Models
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2024Grounding Language Models for Visual Entity Recognition
  6. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Model
  7. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  8. 2023LOIS: Looking Out of Instance Semantics for Visual Question Answering