mlsanityData to 7 Oct 2026

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

arXiv 2304.10592 ↗First posted 20 Apr 2023cs.CVcited by 2,689

Not on the map. Why?

Cited by 1,478 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.

Key facts. Cited by 2,689 papers in the map (data through 7 Oct 2026). Builds on 38 of its 62 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

38 of 62 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
26 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  13. 2018Object Hallucination in Image Captioningcited by 661
  14. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  16. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  17. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  18. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  19. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  20. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  21. 2019Connecting Vision and Language with Localized Narrativescited by 225
  22. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  23. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  24. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  25. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  26. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 467; 2024: 1030; 2025: 830; 2026: 362.

2,689 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
38 more
  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  14. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  15. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  16. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  17. 2023A Survey on Multimodal Large Language Modelscited by 898
  18. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  20. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  21. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  22. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  23. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  24. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  25. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  26. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  27. 2025Qwen2.5-Omni Technical Reportcited by 704
  28. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  29. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  30. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  31. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  32. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  33. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  34. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  35. 2023A Comprehensive Overview of Large Language Modelscited by 639
  36. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  37. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  38. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617

50 most cited of 2,689 All 2,689 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter
  2. 2023Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond
  3. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning
  4. 2023GPT4Image: Large Pre-trained Models Help Vision Models Learn Better on Perception Task
  5. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
  6. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
  7. 2023Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts
  8. 2023Multimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoning