mlsanityData to 14 Oct 2026

Large Multimodal Models: Notes on CVPR 2023 Tutorial

arXiv 2306.14895 ↗First posted 26 Jun 2023cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This tutorial note summarizes the presentation on ``Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4'', a part of CVPR 2023 tutorial on ``Recent Advances in Vision Foundation Models''. The tutorial consists of three parts. We first introduce the background on recent GPT-like large models for vision-and-language modeling to motivate the research in instruction-tuned large multimodal models (LMMs). As a pre-requisite, we describe the basics of instruction-tuning in large language models, which is further extended to the multimodal space. Lastly, we illustrate how to build the minimum prototype of multimodal GPT-4 like models with the open-source resource, and review the recently emerged topics.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 65 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

55 of 65 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  2. 2023Generating Images with Multimodal Language Models26 May
  3. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  12. 2023Instruction Tuning with GPT-4cited by 686
  13. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  17. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  18. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  19. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  20. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  21. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  23. 2023The False Promise of Imitating Proprietary LLMscited by 263
  24. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  25. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  26. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  27. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  28. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  29. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  30. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  31. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 101
  32. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  33. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 90
  34. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 66
  35. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathologycited by 57
  36. 2022Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognitioncited by 45
  37. 2023Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViTcited by 14
  38. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Modelscited by 12

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 9; 2025: 2.

18 papers.

Sort:

By area:

  1. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 256
  3. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  4. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 36
  5. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Modelscited by 32
  6. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  7. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 23
  8. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Frameworkcited by 20
  9. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Modelscited by 16
  10. 2024X-Former: Unifying Contrastive and Reconstruction Learning for MLLMscited by 14
  11. 2024LLaVA-Read: Enhancing Reading Ability of Multimodal Language Modelscited by 12
  12. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9
6 more
  1. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  2. 2024TRINS: Towards Multimodal Language Models that Can Readcited by 8
  3. 2024MMR: Evaluating Reading Ability of Large Multimodal Modelscited by 6
  4. 2024Explaining latent representations of generative models with large multimodal modelscited by 3
  5. 2024Multimodal Rationales for Explainable Visual Question Answeringcited by 2
  6. 2023Concept-Oriented Deep Learning with Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistants
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
  3. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
  4. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2024VisionGPT-3D: A Generalized Multimodal Agent for Enhanced 3D Vision Understanding
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling