mlsanityData to 14 Oct 2026

The Role of Data Curation in Image Captioning

arXiv 2305.03610 ↗First posted 5 May 2023cs.CV cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Image captioning models are typically trained by treating all samples equally, neglecting to account for mismatched or otherwise difficult data points. In contrast, recent work has shown the effectiveness of training models by scheduling the data using curriculum learning strategies. This paper contributes to this direction by actively curating difficult samples in datasets without increasing the total number of samples. We explore the effect of using three data curation methods within the training process: complete removal of an sample, caption replacement, or image replacement via a text-to-image generation model. Experiments on the Flickr30K and COCO datasets with the BLIP and BEiT-3 models demonstrate that these curation methods do indeed yield improved image captioning models, underscoring their efficacy.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

28 of 52 references.

References · 28

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
16 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  3. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  4. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  5. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  6. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  7. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  8. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  9. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 76
  10. 2022Data Curation Alone Can Stabilize In-context Learningcited by 59
  11. 2021Changing the World by Changing the Datacited by 51
  12. 2016Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measurescited by 36
  13. 2021Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioningcited by 27
  14. 2023Multimodal Data Augmentation for Image Captioning using Diffusion Modelscited by 6
  15. 2017Room for improvement in automatic image description: an error analysiscited by 4
  16. 2022Cross-Modal Similarity-Based Curriculum Learning for Image Captioningcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generationcited by 41
  2. 2024Altogether: Image Captioning via Re-aligning Alt-textcited by 17
  3. 2025OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Modelscited by 2
  4. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 1