mlsanityData to 14 Oct 2026

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

arXiv 2311.12793 ↗First posted 21 Nov 2023cs.CVcited by 1,005

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data. To address this bottleneck, we introduce the ShareGPT4V dataset, a pioneering large-scale resource featuring 1.2 million highly descriptive captions, which surpasses existing datasets in diversity and information content, covering world knowledge, object properties, spatial relationships, and aesthetic evaluations. Specifically, ShareGPT4V originates from a curated 100K high-quality captions collected from advanced GPT4-Vision and has been expanded to 1.2M with a superb caption model trained on this subset. ShareGPT4V first demonstrates its effectiveness for the Supervised Fine-Tuning (SFT) phase, by substituting an equivalent quantity of detailed captions in existing SFT datasets with a subset of our high-quality captions, significantly enhancing the LMMs like LLaVA-7B, LLaVA-1.5-13B, and Qwen-VL-Chat-7B on the MME and MMBench benchmarks, with respective gains of 222.8/22.0/22.3 and 2.7/1.3/1.5. We further incorporate ShareGPT4V data into both the pre-training and SFT phases, obtaining ShareGPT4V-7B, a superior LMM based on a simple architecture that has remarkable performance across a majority of the multi-modal benchmarks. This project is available at https://ShareGPT4V.github.io to serve as a pivotal resource for advancing the LMMs community.

Key facts. Cited by 1,005 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

51 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2023Segment Anythingcited by 8,234
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
31 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2021Grounded Language-Image Pre-trainingcited by 1,156
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  15. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  16. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  18. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  19. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  20. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  21. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  22. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  23. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  24. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  25. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  26. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  27. 2023Improving CLIP Training with Language Rewritescited by 217
  28. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
  29. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 101
  30. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  31. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 425; 2025: 381; 2026: 189.

1,005 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  9. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  10. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  11. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
38 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 719
  2. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  4. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  5. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  6. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  8. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  9. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  10. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  12. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  13. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  14. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  15. 2024OmniGen: Unified Image Generationcited by 394
  16. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  17. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  18. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  19. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 329
  20. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  21. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  22. 2024What matters when building vision-language models?cited by 326
  23. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  24. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  25. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  26. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  27. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  28. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  29. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  30. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  31. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  32. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  33. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  34. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  35. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  36. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  37. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  38. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211

50 most cited of 1,005 All 1,005 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Cross-Modal Consistency in Multimodal Large Language Models
  2. 2026Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning
  3. 2023CapsFusion: Rethinking Image-Text Data at Scale
  4. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  5. 2024Can Large Multimodal Models Uncover Deep Semantics Behind Images?
  6. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
  7. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  8. 2024Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis