mlsanityData to 14 Oct 2026

A survey on knowledge-enhanced multimodal learning

arXiv 2211.12328 ↗First posted 19 Nov 2022cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Multimodal learning has been a field of increasing interest, aiming to combine various modalities in a single joint representation. Especially in the area of visiolinguistic (VL) learning multiple models and techniques have been developed, targeting a variety of tasks that involve images and text. VL models have reached unprecedented performances by extending the idea of Transformers, so that both modalities can learn from each other. Massive pre-training procedures enable VL models to acquire a certain level of real-world understanding, although many gaps can be identified: the limited comprehension of commonsense, factual, temporal and other everyday knowledge aspects questions the extendability of VL tasks. Knowledge graphs and other knowledge sources can fill those gaps by explicitly providing missing information, unlocking novel capabilities of VL models. In the same time, knowledge graphs enhance explainability, fairness and validity of decision making, issues of outermost importance for such complex implementations. The current survey aims to unify the fields of VL representation learning and knowledge graphs, and provides a taxonomy and analysis of knowledge-enhanced VL models.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 187 of its 272 references that are in the map. First posted on arXiv 19 Nov 2022.

Builds on

187 of 272 references.

References · 187

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  10. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  12. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
38 more
  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2017Graph Attention Networkscited by 8,923
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  13. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  14. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  15. 2016Improved Techniques for Training GANscited by 2,970
  16. 2015Fast R-CNNcited by 2,933
  17. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  18. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  19. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  20. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  21. 2015VQA: Visual Question Answeringcited by 2,506
  22. 2018Deep contextualized word representationscited by 2,290
  23. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  24. 2014Conditional Generative Adversarial Netscited by 2,162
  25. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  26. 2016node2vec: Scalable Feature Learning for Networkscited by 1,918
  27. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  28. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  29. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  30. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  31. 2014DeepWalk: Online Learning of Social Representationscited by 1,547
  32. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  33. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  34. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  35. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  36. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  37. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  38. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203

50 most cited of 187 All 187 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 2; 2026: 1.

11 papers.

  1. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  2. 2023An Overview Of Temporal Commonsense Reasoning and Acquisitioncited by 10
  3. 2024Beyond DAGs: A Latent Partial Causal Model for Multimodal Learningcited by 10
  4. 2023Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguationcited by 8
  5. 2024Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memescited by 8
  6. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  7. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 1
  8. 2025Robust Symbolic Reasoning for Visual Narratives via Hierarchical and Semantically Normalized Knowledge Graphscited by 1
  9. 2023Knowledge-Based Counterfactual Queries for Visual Question Answeringcited by 0
  10. 2023Lattice-preserving ALC ontology embeddings with saturationcited by 0
  11. 2026M-ArtAgent: Evidence-Based Multimodal Agent for Implicit Art Influence Discoverycited by 0