mlsanityData to 7 Oct 2026

Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models

arXiv 2109.03892 ↗First posted 8 Sep 2021cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

34 of 49 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
22 more
  1. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  2. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  3. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  5. 2016Self-critical Sequence Training for Image Captioningcited by 544
  6. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  7. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  8. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  9. 2020COSMIC: COmmonSense knowledge for eMotion Identification in Conversationscited by 135
  10. 2020KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoningcited by 106
  11. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  12. 2018Discriminability objective for training descriptive captionscited by 67
  13. 2020GenAug: Data Augmentation for Finetuning Text Generatorscited by 41
  14. 2018Findings of the E2E NLG Challengecited by 28
  15. 2020An Enhanced Knowledge Injection Model for Commonsense Generationcited by 21
  16. 2019Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchangecited by 19
  17. 2021Retrieval Enhanced Model for Commonsense Generationcited by 18
  18. 2019A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Modelscited by 17
  19. 2019ALOHA: Artificial Learning of Human Attributes for Dialogue Agentscited by 16
  20. 2021NAREOR: The Narrative Reordering Problemcited by 14
  21. 2020Denoising Pre-Training and Data Augmentation Strategies for Enhanced RDF Verbalization with Transformerscited by 10
  22. 2021SAPPHIRE: Approaches for Enhanced Concept-to-Text Generationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 0; 2024: 1; 2025: 1; 2026: 2.

9 papers.

  1. 2021Contextualized Scene Imagination for Generative Commonsense Reasoningcited by 7
  2. 2022CHARD: Clinical Health-Aware Reasoning Across Dimensions for Text Generation Modelscited by 7
  3. 2022PANCETTA: Phoneme Aware Neural Completion to Elicit Tongue Twisters Automaticallycited by 6
  4. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  5. 2026HalluWorld: A Controlled Benchmark for Hallucination via Reference World Modelscited by 2
  6. 2022PINEAPPLE: Personifying INanimate Entities by Acquiring Parallel Personification data for Learning Enhanced generationcited by 1
  7. 2022ConceptNet infused DialoGPT for Underlying Commonsense Understanding and Reasoning in Dialogue Response Generationcited by 1
  8. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrievalcited by 1
  9. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0