mlsanityData to 7 Oct 2026

What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?

arXiv 2307.02469 ↗First posted 5 Jul 2023cs.CV cs.CLcited by 64

Not on the map. Why?

Cited by 45 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) such as GPT4 have displayed exceptional multi-modal capabilities in following open-ended instructions given images. However, the performance of these models heavily relies on design choices such as network structures, training data, and training strategies, and these choices have not been extensively discussed in the literature, making it difficult to quantify progress in this field. To address this issue, this paper presents a systematic and comprehensive study, quantitatively and qualitatively, on training such models. We implement over 20 variants with controlled settings. Concretely, for network structures, we compare different LLM backbones and model designs. For training data, we investigate the impact of data and sampling strategies. For instructions, we explore the influence of diversified prompts on the instruction-following ability of the trained models. For benchmarks, we contribute the first, to our best knowledge, comprehensive evaluation set including both image and video tasks through crowd-sourcing. Based on our findings, we present Lynx, which performs the most accurate multi-modal understanding while keeping the best multi-modal generation ability compared to existing open-sourced GPT4-style models.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 81 of its 117 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

81 of 117 references.

References · 81

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2016Layer Normalizationcited by 5,579
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  13. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  14. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  15. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  16. 2019Towards VQA Models That Can Readcited by 1,696
  17. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  18. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  19. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  20. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  21. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  22. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  23. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  24. 2023PaLM 2 Technical Reportcited by 1,154
  25. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  26. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  27. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  28. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  29. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  30. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  31. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  32. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  33. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  34. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  35. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  36. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  37. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  38. 2023Instruction Tuning with GPT-4cited by 676

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 26; 2025: 11; 2026: 1.

64 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  5. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  6. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  7. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  9. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  10. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  11. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
38 more
  1. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  2. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  3. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  4. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  5. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  6. 2023GraphLLM: Boosting Graph Reasoning Ability of Large Language Modelcited by 105
  7. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  8. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  9. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  10. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  11. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  12. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoningcited by 64
  13. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  14. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  15. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  16. 2024Towards Vision-Language Geo-Foundation Model: A Surveycited by 36
  17. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  18. 2024DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentationcited by 30
  19. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  20. 2023ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sensecited by 27
  21. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  22. 2025Simple o3: Towards Interleaved Vision-Language Reasoningcited by 24
  23. 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
  24. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  25. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 19
  26. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Taskscited by 18
  27. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  28. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  29. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  30. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  31. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarkscited by 12
  32. 2024Model Composition for Multimodal Large Language Modelscited by 12
  33. 2024From Image to Video, what do we need in multimodal LLMs?cited by 10
  34. 2025Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detectioncited by 8
  35. 2023AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Frameworkcited by 6
  36. 2025Large Language Models as Computable Approximations to Solomonoff Inductioncited by 6
  37. 2023Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Modelscited by 5
  38. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Modelscited by 5

50 most cited of 64 All 64 (JSON)