mlsanityData to 14 Oct 2026

CapsFusion: Rethinking Image-Text Data at Scale

arXiv 2310.20550 ↗First posted 31 Oct 2023cs.CV cs.AI cs.CLcited by 90

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large multimodal models demonstrate remarkable generalist ability to perform diverse multimodal tasks in a zero-shot manner. Large-scale web-based image-text pairs contribute fundamentally to this success, but suffer from excessive noise. Recent studies use alternative captions synthesized by captioning models and have achieved notable benchmark performance. However, our experiments reveal significant Scalability Deficiency and World Knowledge Loss issues in models trained with synthetic captions, which have been largely obscured by their initial benchmark success. Upon closer examination, we identify the root cause as the overly-simplified language structure and lack of knowledge details in existing synthetic captions. To provide higher-quality and more scalable multimodal pretraining data, we propose CapsFusion, an advanced framework that leverages large language models to consolidate and refine information from both web-based image-text pairs and synthetic captions. Extensive experiments show that CapsFusion captions exhibit remarkable all-round superiority over existing captions in terms of model performance (e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample efficiency (requiring 11-16 times less computation than baselines), world knowledge depth, and scalability. These effectiveness, efficiency and scalability advantages position CapsFusion as a promising candidate for future scaling of LMM training.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 71 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

53 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization9 Sep
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr

Other references · 50

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
38 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  9. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  11. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  12. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  13. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  14. 2023Generating Images with Multimodal Language Modelscited by 260
  15. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  16. 2023Improving CLIP Training with Language Rewritescited by 217
  17. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  18. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  19. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 170
  20. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  21. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  22. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  23. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  24. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  25. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  26. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 102
  27. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  28. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 80
  29. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70
  30. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  31. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  32. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 55
  33. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  34. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  35. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  36. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokenscited by 15
  37. 2023DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attentioncited by 10
  38. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 43; 2025: 32; 2026: 14.

90 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  3. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  4. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  5. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  6. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  7. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  8. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 146
  9. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  10. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  11. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  12. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
38 more
  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  2. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
  3. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  4. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  5. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 73
  6. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 70
  7. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  8. 2024Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesiscited by 56
  9. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 55
  10. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  11. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  12. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  13. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 50
  14. 2024DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detectioncited by 48
  15. 2024Scalable Image Tokenization with Index Backpropagation Quantizationcited by 44
  16. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  17. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  18. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  19. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 39
  20. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  21. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  22. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
  23. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  24. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  25. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 30
  26. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  27. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  28. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  29. 2024RWKV-CLIP: A Robust Vision-Language Representation Learnercited by 23
  30. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  31. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 22
  32. 2024Visual Perception by Large Language Model's Weightscited by 19
  33. 2024MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modelingcited by 19
  34. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  35. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  36. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  37. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  38. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models
  2. 2024Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
  3. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  4. 2023Image Captioning with Multi-Context Synthetic Data
  5. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
  6. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  7. 2026Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling
  8. 2026MultiModal Fine-tuning with Synthetic Captions