mlsanityData to 14 Oct 2026

BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions

arXiv 2411.07461 ↗First posted 12 Nov 2024cs.CV cs.AIcited by 9

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text. KALE augments synthetic dense image captions with web-scale alt-text to generate factually grounded image captions. Our two-stage approach leverages large vision-language models and language models to create knowledge-augmented captions, which are then used to train a specialized VLM for scaling up the dataset. We train vision-language models on KALE and demonstrate improvements on vision-language tasks. Our experiments show the utility of KALE for training more capable and knowledgeable multimodal models. We release the KALE dataset at https://huggingface.co/datasets/Salesforce/blip3-kale

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 33 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

29 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024What matters when building vision-language models?3 May
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  3. 2023CapsFusion: Rethinking Image-Text Data at Scale31 Oct
  4. 2023Improving Multimodal Datasets with Image Captioning19 Jul
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 24

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2016A Diagram Is Worth A Dozen Imagescited by 804
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  12. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
12 more
  1. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  2. 2021InfographicVQAcited by 437
  3. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  4. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  5. 2023Data Filtering Networkscited by 250
  6. 2023Improving CLIP Training with Language Rewritescited by 217
  7. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  8. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  9. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  10. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70
  11. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  12. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 2.

9 papers.

  1. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  2. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  3. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  4. 2025TikZero: Zero-Shot Text-Guided Graphics Program Synthesiscited by 24
  5. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 20
  6. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMscited by 7
  7. 2026Claim-Level Rubric Rewards for Video Caption Reinforcement Learningcited by 3
  8. 2025OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Modelscited by 2
  9. 2026Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
  2. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions
  3. 2024SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models
  4. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
  5. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World
  6. 2024Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models
  7. 2025Text-Only Data Synthesis for Vision Language Model Training
  8. 2026Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling