mlsanityData to 14 Oct 2026

RedCaps: web-curated image-text data created by the people, for the people

arXiv 2111.11431 ↗First posted 22 Nov 2021cs.CV cs.CLcited by 180

Not on the map. Why?

Cited by 38 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large datasets of paired images and text have become increasingly popular for learning generic representations for vision and vision-and-language tasks. Such datasets have been built by querying search engines or collecting HTML alt-text -- since web data is noisy, they require complex filtering pipelines to maintain quality. We explore alternate data sources to collect high quality data with minimal filtering. We introduce RedCaps -- a large-scale dataset of 12M image-text pairs collected from Reddit. Images and captions from Reddit depict and describe a wide variety of objects and scenes. We collect data from a manually curated set of subreddits, which give coarse image labels and allow us to steer the dataset composition without labeling individual instances. We show that captioning models trained on RedCaps produce rich and varied captions preferred by humans, and learn visual representations that transfer to many downstream tasks.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). Builds on 70 of its 119 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

70 of 119 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2017Mask R-CNNcited by 7,277
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  13. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  14. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  15. 2018Datasheets for Datasetscited by 1,386
  16. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  17. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  19. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  21. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  22. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  23. 2020On Layer Normalization in the Transformer Architecturecited by 922
  24. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  25. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  26. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  27. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  28. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  29. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  30. 2018nocaps: novel object captioning at scalecited by 444
  31. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  32. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  33. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  34. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  35. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  36. 2020The Pushshift Reddit Datasetcited by 292
  37. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  38. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 24; 2023: 48; 2024: 52; 2025: 31; 2026: 24.

180 papers.

Sort:

By area:

  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  5. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  8. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  9. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  10. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  11. 2024What matters when building vision-language models?cited by 326
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
38 more
  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  2. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  3. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  4. 2023Improving CLIP Training with Language Rewritescited by 217
  5. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  6. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  7. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  8. 2023Power Hungry Processing: Watts Driving the Cost of AI Deployment?cited by 191
  9. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  10. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  11. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  12. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 155
  13. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  14. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 151
  15. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  16. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 137
  17. 2023Hyperbolic Image-Text Representationscited by 136
  18. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  19. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  20. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  21. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  22. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  23. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  24. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  25. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  26. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  27. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  28. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  29. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  30. 2024Compositional Entailment Learning for Hyperbolic Vision-Language Modelscited by 74
  31. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  32. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  33. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  34. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
  35. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
  36. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  37. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 52
  38. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 48

50 most cited of 180 All 180 (JSON)