mlsanityData to 14 Oct 2026

Guided Open Vocabulary Image Captioning with Constrained Beam Search

arXiv 1612.00576 ↗First posted 2 Dec 2016cs.CVcited by 105

Not on the map. Why?

Abstract (arXiv)

Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images in the wild. We address this problem using a flexible approach that enables existing deep captioning architectures to take advantage of image taggers at test time, without re-training. Our method uses constrained beam search to force the inclusion of selected tag words in the output, and fixed, pretrained word embeddings to facilitate vocabulary expansion to previously unseen tag words. Using this approach we achieve state of the art results for out-of-domain captioning on MSCOCO (and improved results for in-domain captioning). Perhaps surprisingly, our results significantly outperform approaches that incorporate the same tag predictions into the learning algorithm. We also show that we can significantly improve the quality of generated ImageNet captions by leveraging ground-truth labels.

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 18; 2023: 30; 2024: 20; 2025: 7; 2026: 7.

105 papers.

Sort:

By area:

  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  2. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  3. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  4. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
  5. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  6. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 163
  7. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 154
  8. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  9. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  10. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 129
  11. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 88
  12. 2023Controlled Text Generation with Natural Language Instructionscited by 87
38 more
  1. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  2. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  3. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  4. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  5. 2024Grammar-Aligned Decodingcited by 54
  6. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 46
  7. 2022LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restrictioncited by 43
  8. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  9. 2022Controllable Text Generation with Neurally-Decomposed Oraclecited by 41
  10. 2024Constrained Decoding for Secure Code Generationcited by 37
  11. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  12. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  13. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  14. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  15. 2024Permute-and-Flip: An optimally stable and watermarkable decoder for LLMscited by 22
  16. 2022MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENerationcited by 20
  17. 2022Why is constrained neural language generation particularly challenging?cited by 19
  18. 2023Zero-Resource Hallucination Prevention for Large Language Modelscited by 19
  19. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  20. 2022Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioningcited by 18
  21. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  22. 2021On the Evaluation of Machine Translation for Terminology Consistencycited by 15
  23. 2023Formally Specifying the High-Level Behavior of LLM-Based Agentscited by 15
  24. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
  25. 2023Zero-shot Clarifying Question Generation for Conversational Searchcited by 14
  26. 2025Constrained Sampling for Language Models Should Be Easy: An MCMC Perspectivecited by 14
  27. 2023Don't Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decodingcited by 13
  28. 2021Prix-LM: Pretraining for Multilingual Knowledge Base Constructioncited by 12
  29. 2023PathFinder: Guided Search over Multi-Step Reasoning Pathscited by 12
  30. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  31. 2023Model-Based Minimum Bayes Risk Decoding for Text Generationcited by 11
  32. 2022NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledgecited by 10
  33. 2024CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Taskscited by 10
  34. 2025Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Modelscited by 10
  35. 2023GLUECons: A Generic Benchmark for Learning Under Constraintscited by 9
  36. 2023EXPLAIN, EDIT, GENERATE: Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verificationcited by 9
  37. 2023User Adaptive Language Learning Chatbots with a Curriculumcited by 8
  38. 2023A Lightweight Constrained Generation Alternative for Query-focused Summarizationcited by 8

50 most cited of 105 All 105 (JSON)