mlsanityData to 14 Oct 2026

Improved Image Captioning via Policy Gradient optimization of SPIDEr

arXiv 1612.00370 ↗First posted 1 Dec 2016cs.CV cs.CLcited by 113

Not on the map. Why?

Abstract (arXiv)

Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 0; 2020: 1; 2021: 20; 2022: 23; 2023: 19; 2024: 22; 2025: 14; 2026: 13.

113 papers.

Sort:

By area:

  1. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  3. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  4. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  5. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  6. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  7. 2023Audio-Visual LLM for Video Understandingcited by 77
  8. 2023Learning to Generate Better Than Your LLMcited by 49
  9. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  10. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  11. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  12. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 33
38 more
  1. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  2. 2024Taming Data and Transformers for Audio Generationcited by 28
  3. 2021Towards Accurate Text-based Image Captioning with Content Diversity Explorationcited by 25
  4. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  5. 2021Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learningcited by 24
  6. 2023BOLT: Fast Energy-based Controlled Text Generation with Tunable Biasescited by 24
  7. 2021MusCaps: Generating Captions for Music Audiocited by 23
  8. 2022Automated Audio Captioning: An Overview of Recent Progress and New Challengescited by 23
  9. 2021CIDEr-R: Robust Consensus-based Image Description Evaluationcited by 22
  10. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  11. 2017Reinforced Video Captioning with Entailment Rewardscited by 21
  12. 2022Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Taskscited by 21
  13. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  14. 2021SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysiscited by 18
  15. 2022Leveraging Pre-trained BERT for Audio Captioningcited by 18
  16. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  17. 2021Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generationcited by 17
  18. 2024Learning Spatially-Aware Language and Audio Embeddingscited by 17
  19. 2021Contextualized Keyword Representations for Multi-modal Retinal Image Captioningcited by 16
  20. 2021Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"cited by 16
  21. 2021CL4AC: A Contrastive Loss for Audio Captioningcited by 16
  22. 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
  23. 2021Reinforcement Learning-powered Semantic Communication via Semantic Similaritycited by 15
  24. 2022Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learningcited by 15
  25. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14
  26. 2024Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encodingcited by 14
  27. 2021A Survey of Natural Language Generationcited by 13
  28. 2021The Role of Syntactic Planning in Compositional Image Captioningcited by 12
  29. 2021Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularizationcited by 12
  30. 2022Exploring Train and Test-Time Augmentations for Audio-Language Learningcited by 11
  31. 2023Zero-shot audio captioning with audio-language model guidance and audio context keywordscited by 11
  32. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioningcited by 11
  33. 2022Show, Interpret and Tell: Entity-aware Contextualised Image Captioning in Wikipediacited by 10
  34. 2023DRL4Route: A Deep Reinforcement Learning Framework for Pick-up and Delivery Route Predictioncited by 10
  35. 2022Rethinking the Reference-based Distinctive Image Captioningcited by 8
  36. 2022Semantics-Empowered Communication: A Tutorial-cum-Surveycited by 8
  37. 2023C3Net: Compound Conditioned ControlNet for Multimodal Content Generationcited by 8
  38. 2024Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commentingcited by 8

50 most cited of 113 All 113 (JSON)