mlsanityData to 14 Oct 2026

Altogether: Image Captioning via Re-aligning Alt-text

arXiv 2410.17251 ↗First posted 22 Oct 2024cs.CV cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ignoring existing alt-text metadata, and second, lack transparency if the captioners' training data (e.g. GPT) is unknown. In this paper, we study a principled approach Altogether based on the key idea to edit and re-align existing alt-texts associated with the images. To generate training data, we perform human annotation where annotators start with the existing alt-text and re-align it to the image content in multiple rounds, consequently constructing captions with rich visual concepts. This differs from prior work that carries out human annotation as a one-time description task solely based on images and annotator knowledge. We train a captioner on this data that generalizes the process of re-aligning alt-texts at scale. Our results show our Altogether approach leads to richer image captions that also improve text-to-image generation and zero-shot image classification tasks.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

22 of 32 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  10. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  11. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  12. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
10 more
  1. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  2. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  3. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  4. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  5. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  6. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70
  7. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
  8. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  9. 2023The Role of Data Curation in Image Captioningcited by 4
  10. 2020Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 3.

17 papers.

Sort:

By area:

  1. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  2. 2025Scaling Language-Free Visual Representation Learningcited by 66
  3. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 34
  4. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  5. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  6. 2025On the Value of Cross-Modal Misalignment in Multimodal Representation Learningcited by 10
  7. 2025Brought a Gun to a Knife Fight: Modern VFM Baselines Outgun Specialized Detectors on In-the-Wild AI Image Detectioncited by 10
  8. 2025GAIA: A Global, Multi-modal, Multi-scale Vision-Language Dataset for Remote Sensing Image Analysiscited by 7
  9. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  10. 2025RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstructioncited by 4
  11. 2025RoboSVG: A Unified Framework for Interactive SVG Generation with Multi-modal Guidancecited by 2
  12. 2025Caption Injection for Optimization in Generative Search Enginecited by 2
5 more
  1. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learningcited by 2
  2. 2025Using Knowledge Graphs to harvest datasets for efficient CLIP model trainingcited by 0
  3. 2025Probing the Limits of Stylistic Alignment in Vision-Language Modelscited by 0
  4. 2026Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioningcited by 0
  5. 2026EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Datacited by 0