mlsanityData to 7 Oct 2026

MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning

arXiv 2310.09478 ↗First posted 14 Oct 2023cs.CVcited by 617

Not on the map. Why?

Cited by 398 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have shown their remarkable capabilities as a general interface for various language-related applications. Motivated by this, we target to build a unified interface for completing many vision-language tasks including image description, visual question answering, and visual grounding, among others. The challenge is to use a single model for performing diverse vision-language tasks effectively with simple multi-modal instructions. Towards this objective, we introduce MiniGPT-v2, a model that can be treated as a unified interface for better handling various vision-language tasks. We propose using unique identifiers for different tasks when training the model. These identifiers enable our model to better distinguish each task instruction effortlessly and also improve the model learning efficiency for each task. After the three-stage training, the experimental results show that MiniGPT-v2 achieves strong performance on many visual question-answering and visual grounding benchmarks compared to other vision-language generalist models. Our model and codes are available at https://minigpt-v2.github.io/

Key facts. Cited by 617 papers in the map (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 14 Oct 2023.

Builds on

49 of 65 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
37 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  14. 2016Modeling Context in Referring Expressionscited by 1,054
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  17. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  18. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  19. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  21. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  22. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  23. 2018Object Hallucination in Image Captioningcited by 661
  24. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  25. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  26. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  27. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  28. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  29. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  30. 2022Visual Spatial Reasoningcited by 288
  31. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  32. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  33. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  34. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  35. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  36. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  37. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 50; 2024: 313; 2025: 192; 2026: 62.

617 papers.

Sort:

By area:

  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  5. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  6. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  8. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  9. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  10. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
38 more
  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  4. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  7. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  8. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  9. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  10. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  11. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  12. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  13. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  14. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  15. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  16. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  17. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  18. 2024An Introduction to Vision-Language Modelingcited by 175
  19. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  20. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  21. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  22. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  23. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  24. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  25. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  26. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  27. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  28. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  29. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  30. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  31. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  32. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  33. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  34. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  35. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  36. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  37. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  38. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124

50 most cited of 617 All 617 (JSON)