mlsanityData to 7 Oct 2026

Self-critical Sequence Training for Image Captioning

arXiv 1612.00563 ↗First posted 2 Dec 2016cs.LG cs.AI cs.CVcited by 544

Not on the map. Why?

Cited by 40 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently it has been shown that policy-gradient methods for reinforcement learning can be utilized to train deep end-to-end systems directly on non-differentiable metrics for the task at hand. In this paper we consider the problem of optimizing image captioning systems using reinforcement learning, and show that by carefully optimizing our systems using the test metrics of the MSCOCO task, significant gains in performance can be realized. Our systems are built using a new optimization approach that we call self-critical sequence training (SCST). SCST is a form of the popular REINFORCE algorithm that, rather than estimating a "baseline" to normalize the rewards and reduce variance, utilizes the output of its own test-time inference algorithm to normalize the rewards it experiences. Using this approach, estimating the reward signal (as actor-critic methods must do) and estimating normalization (as REINFORCE algorithms typically do) is avoided, while at the same time harmonizing the model with respect to its test-time inference procedure. Empirically we find that directly optimizing the CIDEr metric with SCST and greedy decoding at test-time is highly effective. Our results on the MSCOCO evaluation sever establish a new state-of-the-art on the task, improving the best result in terms of CIDEr from 104.9 to 114.7.

Key facts. Cited by 544 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 0; 2019: 1; 2020: 2; 2021: 131; 2022: 137; 2023: 116; 2024: 68; 2025: 48; 2026: 40.

544 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  7. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  9. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  10. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  11. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  12. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
38 more
  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  3. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  4. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  5. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  6. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  7. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  8. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  9. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  10. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  11. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  12. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  13. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  14. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  15. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  16. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  17. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  18. 2021Causal Attention for Vision-Language Taskscited by 111
  19. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  20. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  21. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  22. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  23. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  24. 2022Language Models are General-Purpose Interfacescited by 96
  25. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  26. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 93
  27. 2023Input Perturbation Reduces Exposure Bias in Diffusion Modelscited by 88
  28. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  29. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  30. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 82
  31. 2021Protecting Intellectual Property of Language Generation APIs with Lexical Watermarkcited by 78
  32. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  33. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  34. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 74
  35. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  36. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  37. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  38. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70

50 most cited of 544 All 544 (JSON)