mlsanityData to 14 Oct 2026

Neural Baby Talk

arXiv 1803.09845 ↗First posted 27 Mar 2018cs.CV cs.CLcited by 107

Not on the map. Why?

Abstract (arXiv)

We introduce a novel framework for image captioning that can produce natural language explicitly grounded in entities that object detectors find in the image. Our approach reconciles classical slot filling approaches (that are generally better grounded in images) with modern neural captioning approaches (that are generally more natural sounding and accurate). Our approach first generates a sentence `template' with slot locations explicitly tied to specific image regions. These slots are then filled in by visual concepts identified in the regions by object detectors. The entire architecture (sentence template generation and slot filling with object detectors) is end-to-end differentiable. We verify the effectiveness of our proposed model on different image captioning tasks. On standard image captioning and novel object captioning, our model reaches state-of-the-art on both COCO and Flickr30k datasets. We also demonstrate that our model has unique advantages when the train and test distributions of scene compositions -- and hence language priors of associated captions -- are different. Code has been made available at: https://github.com/jiasenlu/NeuralBabyTalk

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 30; 2023: 27; 2024: 9; 2025: 8; 2026: 2.

107 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
  5. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  6. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  7. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  8. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  9. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  10. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  11. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  12. 2021ArtEmis: Affective Language for Visual Artcited by 104
38 more
  1. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  2. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  3. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72
  4. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  5. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  6. 2023AutoAD: Movie Description in Contextcited by 50
  7. 2021Towards General Purpose Vision Systemscited by 46
  8. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 46
  9. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  10. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  11. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
  12. 2022It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collectioncited by 36
  13. 2021Zero-shot Natural Language Video Localizationcited by 34
  14. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  15. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  16. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  17. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  18. 2021Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generationcited by 30
  19. 2021Consensus Graph Representation Learning for Better Grounded Image Captioningcited by 28
  20. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  21. 2021Towards Accurate Text-based Image Captioning with Content Diversity Explorationcited by 25
  22. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  23. 2021ReFormer: The Relational Transformer for Image Captioningcited by 23
  24. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23
  25. 2023ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embeddingcited by 23
  26. 2021CIDEr-R: Robust Consensus-based Image Description Evaluationcited by 22
  27. 2022Vision Transformers in Medical Computer Vision -- A Contemplative Retrospectioncited by 22
  28. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  29. 2022A Self-Guided Framework for Radiology Report Generationcited by 15
  30. 2023Transforming Visual Scene Graphs to Image Captionscited by 15
  31. 2021Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioningcited by 14
  32. 2021Journalistic Guidelines Aware News Image Captioningcited by 14
  33. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14
  34. 2021Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrievalcited by 13
  35. 2024MetaToken: Detecting Hallucination in Image Descriptions by Meta Classificationcited by 13
  36. 2021The Role of Syntactic Planning in Compositional Image Captioningcited by 12
  37. 2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12
  38. 2021Neural Attention for Image Captioning: Review of Outstanding Methodscited by 11

50 most cited of 107 All 107 (JSON)