mlsanityData to 7 Oct 2026

Word to Sentence Visual Semantic Similarity for Caption Generation: Lessons Learned

arXiv 2209.12817 ↗First posted 26 Sep 2022cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper focuses on enhancing the captions generated by image-caption generation systems. We propose an approach for improving caption generation systems by choosing the most closely related output to the image rather than the most likely output produced by the model. Our model revises the language generation output beam search from a visual context perspective. We employ a visual semantic measure in a word and sentence level manner to match the proper caption to the related information in the image. The proposed approach can be applied to any caption system as a post-processing based method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

28 of 45 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
16 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  3. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  4. 2019Attention on Attention for Image Captioningcited by 289
  5. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  6. 2020X-Linear Attention Networks for Image Captioningcited by 198
  7. 2014From Captions to Visual Concepts and Backcited by 109
  8. 2018Neural Baby Talkcited by 107
  9. 2017Deep Keyphrase Generationcited by 102
  10. 2017SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publicationscited by 101
  11. 2018Discriminability objective for training descriptive captionscited by 67
  12. 2018Show, Control and Tell: A Framework for Generating Controllable and Grounded Captionscited by 65
  13. 2019Describing like humans: on diversity in image captioningcited by 46
  14. 2020More Grounded Image Captioning by Distilling Image-Text Matching Modelcited by 43
  15. 2018Object Counts! Bringing Explicit Detections Back into Image Captioningcited by 5
  16. 2018Visual Re-ranking with Natural Language Understanding for Text Spottingcited by 4

Built on by

Not cited yet.