mlsanityData to 7 Oct 2026

KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation

arXiv 2101.00419 ↗First posted 2 Jan 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

We present Knowledge Enhanced Multimodal BART (KM-BART), which is a Transformer-based sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. We adapt the generative BART architecture to a multimodal model with visual and textual inputs. We further develop novel pretraining tasks to improve the model performance on the Visual Commonsense Generation (VCG) task. In particular, our pretraining task of Knowledge-based Commonsense Generation (KCG) boosts model performance on the VCG task by leveraging commonsense knowledge from a large language model pretrained on external commonsense knowledge graphs. To the best of our knowledge, we are the first to propose a dedicated task for improving model performance on the VCG task. Experimental results show that our model reaches state-of-the-art performance on the VCG task by applying these novel pretraining tasks.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 2 Jan 2021.

Builds on

22 of 31 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2017Mask R-CNNcited by 7,259
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
10 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  2. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  3. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  4. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  5. 2016Image Captioning with Semantic Attentioncited by 260
  6. 2018A Simple Method for Commonsense Reasoningcited by 223
  7. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  8. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  9. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  10. 2018Machine Common Sense Concept Papercited by 17

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 10; 2023: 5; 2024: 3.

19 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  2. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  3. 2022Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysiscited by 21
  4. 2023Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generationcited by 19
  5. 2023UniSA: Unified Generative Framework for Sentiment Analysiscited by 17
  6. 2022Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysiscited by 16
  7. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  8. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  9. 2023Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioningcited by 14
  10. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  11. 2021A Simple But Powerful Graph Encoder for Temporal Knowledge Graph Completioncited by 8
  12. 2022COFAR: Commonsense and Factual Reasoning in Image Searchcited by 4
7 more
  1. 2022DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attentioncited by 4
  2. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  3. 2024Sumotosima: A Framework and Dataset for Classifying and Summarizing Otoscopic Imagescited by 1
  4. 2022UTSA NLP at SemEval-2022 Task 4: An Exploration of Simple Ensembles of Transformers, Convolutional, and Recurrent Neural Networkscited by 0
  5. 2022Attention Mechanism based Cognition-level Scene Understandingcited by 0
  6. 2022A Marker-based Neural Network System for Extracting Social Determinants of Healthcited by 0
  7. 2024DIVE: Towards Descriptive and Diverse Visual Commonsense Generationcited by 0