mlsanityData to 7 Oct 2026

MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

arXiv 2305.04790 ↗First posted 8 May 2023cs.CV cs.CLcited by 255

Not on the map. Why?

Cited by 145 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a vision and language model named MultiModal-GPT to conduct multi-round dialogue with humans. MultiModal-GPT can follow various instructions from humans, such as generating a detailed caption, counting the number of interested objects, and answering general questions from users. MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with Low-rank Adapter (LoRA) added both in the cross-attention part and the self-attention part of the language model. We first construct instruction templates with vision and language data for multi-modality instruction tuning to make the model understand and follow human instructions. We find the quality of training data is vital for the dialogue performance, where few data containing short answers can lead the model to respond shortly to any instructions. To further enhance the ability to chat with humans of the MultiModal-GPT, we utilize language-only instruction-following data to train the MultiModal-GPT jointly. The joint training of language-only and visual-language instructions with the same instruction template effectively improves dialogue performance. Various demos show the ability of continuous dialogue of MultiModal-GPT with humans. Code, dataset, and demo are at https://github.com/open-mmlab/Multimodal-GPT

Key facts. Cited by 255 papers in the map (data through 7 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

14 of 18 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
2 more
  1. 2023Instruction Tuning with GPT-4cited by 676
  2. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 112; 2024: 92; 2025: 43; 2026: 8.

255 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
38 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  9. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  10. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  11. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  12. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  13. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  14. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  15. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  16. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  18. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  19. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  20. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  21. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  22. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  23. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  24. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  25. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmarkcited by 204
  26. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  27. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  28. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  29. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  30. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  31. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  32. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  33. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  34. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  35. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  36. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  37. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  38. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124

50 most cited of 255 All 255 (JSON)