mlsanityData to 7 Oct 2026

CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning

arXiv 2210.04559 ↗First posted 10 Oct 2022cs.CV cs.LGcited by 11

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Image captioning task has been extensively researched by previous work. However, limited experiments focus on generating captions based on non-autoregressive text decoder. Inspired by the recent success of the denoising diffusion model on image synthesis tasks, we apply denoising diffusion probabilistic models to text generation in image captioning tasks. We show that our CLIP-Diffusion-LM is capable of generating image captions using significantly fewer inference steps than autoregressive models. On the Flickr8k dataset, the model achieves 0.1876 BLEU-4 score. By training on the combined Flickr8k and Flickr30k dataset, our model achieves 0.2470 BLEU-4 score. Our code is available at https://github.com/xu-shitong/diffusion-image-captioning.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 36 of its 50 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

36 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2022Generative Adversarial Networkscited by 5,603
21 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  8. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  9. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  10. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  11. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  12. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  13. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  14. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  15. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  16. 2018Unsupervised Image Captioningcited by 79
  17. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  18. 2019Masked Non-Autoregressive Image Captioningcited by 22
  19. 2021Global-Local Attention for Emotion Recognitioncited by 20
  20. 2021Semi-Autoregressive Image Captioningcited by 10
  21. 2019Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generationcited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 6; 2024: 1; 2025: 3.

11 papers.

  1. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  2. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  3. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
  4. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17
  5. 2023Graph Neural Networks in Vision-Language Image Understanding: A Surveycited by 10
  6. 2024LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?cited by 9
  7. 2022Weakly Supervised Annotations for Multi-modal Greeting Cards Datasetcited by 0
  8. 2023Comparative study of Transformer and LSTM Network with attention mechanism on Image Captioningcited by 0
  9. 2023DiffuVST: Narrating Fictional Scenes with Global-History-Guided Denoising Modelscited by 0
  10. 2025FusDreamer: Label-efficient Remote Sensing World Model for Multimodal Data Classificationcited by 0
  11. 2025Diffusion Is Your Friend in Show, Suggest and Tellcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffCap: Exploring Continuous Diffusion on Image Captioning
  2. 2023Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning
  3. 2022Exploring Discrete Diffusion Models for Image Captioning
  4. 2023Controlled and Conditional Text to Image Generation with Diffusion Prior
  5. 2023A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  6. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  7. 2025Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning
  8. 2026ImageRAGTurbo: Towards One-step Text-to-Image Generation with Retrieval-Augmented Diffusion Models