mlsanityData to 7 Oct 2026

From Pixels to Prose: A Large Dataset of Dense Image Captions

arXiv 2406.10328 ↗First posted 14 Jun 2024cs.CV cs.CL cs.LGcited by 52

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Training large vision-language models requires extensive, high-quality image-text pairs. Existing web-scraped datasets, however, are noisy and lack detailed image descriptions. To bridge this gap, we introduce PixelProse, a comprehensive dataset of over 16M (million) synthetically generated captions, leveraging cutting-edge vision-language models for detailed and accurate descriptions. To ensure data integrity, we rigorously analyze our dataset for problematic content, including child sexual abuse material (CSAM), personally identifiable information (PII), and toxicity. We also provide valuable metadata such as watermark presence and aesthetic scores, aiding in further dataset filtering. We hope PixelProse will be a valuable resource for future vision-language research. PixelProse is available at https://huggingface.co/datasets/tomg-group-umd/pixelprose

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 64 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

48 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  2. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  3. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model6 Feb
  4. 2023CapsFusion: Rethinking Image-Text Data at Scale31 Oct

Other references · 44

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2018Datasheets for Datasetscited by 1,371
32 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  4. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  6. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  11. 2018nocaps: novel object captioning at scalecited by 444
  12. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  13. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  14. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  15. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  16. 2019Character Region Awareness for Text Detectioncited by 273
  17. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  18. 2019Connecting Vision and Language with Localized Narrativescited by 225
  19. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  20. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  21. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  22. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  23. 2020Captioning Images Taken by People Who Are Blindcited by 108
  24. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  25. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  26. 2021A Study of Face Obfuscation in ImageNetcited by 83
  27. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuningcited by 73
  28. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  29. 2023Into the LAIONs Den: Investigating Hate in Multimodal Datasetscited by 54
  30. 2021PASS: An ImageNet replacement for self-supervised pretraining without humanscited by 51
  31. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  32. 2023Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Methodcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 27; 2026: 13.

52 papers.

Sort:

By area:

  1. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  2. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  3. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  4. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  5. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  6. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  7. 2025Unified Multimodal Discrete Diffusioncited by 46
  8. 2024One Diffusion to Generate Them Allcited by 41
  9. 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 37
  10. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  11. 2025Transition Models: Rethinking the Generative Learning Objectivecited by 32
  12. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
38 more
  1. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  2. 2024Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registerscited by 19
  3. 2025UGen: Unified Autoregressive Multimodal Model with Progressive Vocabulary Learningcited by 16
  4. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  5. 2024COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-trainingcited by 15
  6. 2025COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generationcited by 12
  7. 2025MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversationscited by 12
  8. 2026RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioningcited by 9
  9. 2025Mitigating Image Captioning Hallucinations in Vision-Language Modelscited by 7
  10. 2024SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Modelscited by 6
  11. 2025VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMscited by 6
  12. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5
  13. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captionscited by 5
  14. 2025What's Producible May Not Be Reachable: Measuring the Steerability of Generative Modelscited by 5
  15. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  16. 2026Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budgetcited by 5
  17. 2024Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluationcited by 4
  18. 2025HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claimscited by 4
  19. 2025Vision-Speech Models: Teaching Speech Models to Converse about Imagescited by 4
  20. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  21. 2025Teamwork: Collaborative Diffusion with Low-rank Coordination and Adaptationcited by 3
  22. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  23. 2025Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVAcited by 1
  24. 2025Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captionscited by 1
  25. 2025The Devil is in the EOS: Sequence Training for Detailed Image Captioningcited by 1
  26. 2025Masked Diffusion Captioning for Visual Feature Learningcited by 1
  27. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  28. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  29. 2026SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generationcited by 1
  30. 2026Doomed to Re-Annotate, Forever: The ImageNet Storycited by 1
  31. 2025Document-Level Text Generation with Minimum Bayes Risk Decoding using Optimal Transportcited by 0
  32. 2025MS-DPPs: Multi-Source Determinantal Point Processes for Contextual Diversity Refinement of Composite Attributes in Text to Image Retrievalcited by 0
  33. 2025ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generationcited by 0
  34. 2026Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioningcited by 0
  35. 2026Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Modelscited by 0
  36. 2026BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Modelcited by 0
  37. 2026MIDAL: A Dataset of Math Image Descriptions for Accessible Learningcited by 0
  38. 2026KwaiMind Technical Reportcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
  2. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  3. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  4. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
  5. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
  6. 2023Improving Multimodal Datasets with Image Captioning
  7. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
  8. 2023Filter & Align: Leveraging Human Knowledge to Curate Image-Text Data