mlsanityData to 7 Oct 2026

Exploring Discrete Diffusion Models for Image Captioning

arXiv 2211.11694 ↗First posted 21 Nov 2022cs.CVcited by 16

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one. We present a diffusion-based captioning model, dubbed the name DDCap, to allow more decoding flexibility. Unlike image generation, where the output is continuous and redundant with a fixed length, texts in image captions are categorical and short with varied lengths. Therefore, naively applying the discrete diffusion model to text decoding does not work well, as shown in our experiments. To address the performance gap, we propose several key techniques including best-first inference, concentrated attention mask, text length prediction, and image-free training. On COCO without additional caption pre-training, it achieves a CIDEr score of 117.8, which is +5.0 higher than the auto-regressive baseline with the same architecture in the controlled setting. It also performs +26.8 higher CIDEr score than the auto-regressive baseline (230.3 v.s.203.5) on a caption infilling task. With 4M vision-language pre-training images and the base-sized model, we reach a CIDEr score of 125.1 on COCO, which is competitive to the best well-developed auto-regressive frameworks. The code is available at https://github.com/buxiangzhiren/DDCap.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 71 of its 81 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

71 of 81 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  6. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  13. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  14. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  15. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  16. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  17. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  18. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  19. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  20. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  21. 2016Self-critical Sequence Training for Image Captioningcited by 544
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  23. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  24. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  25. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  26. 2019Attention on Attention for Image Captioningcited by 289
  27. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  28. 2016Image Captioning with Semantic Attentioncited by 260
  29. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  30. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  31. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  32. 2018Exploring Visual Relationship for Image Captioningcited by 215
  33. 2020X-Linear Attention Networks for Image Captioningcited by 198
  34. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  35. 2019Image Captioning: Transforming Objects into Wordscited by 171
  36. 2022gDDIM: Generalized denoising diffusion implicit modelscited by 125
  37. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  38. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 2; 2025: 3; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  2. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  3. 2023Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --cited by 71
  4. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 56
  5. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 38
  6. 2023DLT: Conditioned layout generation with Joint Discrete-Continuous Diffusion Layout Transformercited by 20
  7. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17
  8. 2023IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognitioncited by 12
  9. 2024LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?cited by 9
  10. 2023Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioningcited by 7
  11. 2023DECap: Towards Generalized Explicit Caption Editing via Diffusion Mechanismcited by 1
  12. 2024Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Imagecited by 1
4 more
  1. 2025Generalized Visual Relation Detection with Diffusion Modelscited by 0
  2. 2025Diffusion Is Your Friend in Show, Suggest and Tellcited by 0
  3. 2025CrossTrafficLLM: A Human-Centric Framework for Interpretable Traffic Intelligence via Large Language Modelcited by 0
  4. 2026Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffCap: Exploring Continuous Diffusion on Image Captioning
  2. 2023Multimodal Data Augmentation for Image Captioning using Diffusion Models
  3. 2022CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning
  4. 2025Masked Diffusion Captioning for Visual Feature Learning
  5. 2022Semantic-Conditional Diffusion Networks for Image Captioning
  6. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  7. 2024Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning
  8. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models