mlsanityData to 14 Oct 2026

Endowing Language Models with Multimodal Knowledge Graph Representations

arXiv 2206.13163 ↗First posted 27 Jun 2022cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

We propose a method to make natural language understanding models more parameter efficient by storing knowledge in an external knowledge graph (KG) and retrieving from this KG using a dense index. Given (possibly multilingual) downstream task data, e.g., sentences in German, we retrieve entities from the KG and use their multimodal representations to improve downstream task performance. We use the recently released VisualSem KG as our external knowledge repository, which covers a subset of Wikipedia and WordNet entities, and compare a mix of tuple-based and graph-based algorithms to learn entity and relation representations that are grounded on the KG multimodal information. We demonstrate the usefulness of the learned entity representations on two downstream tasks, and show improved performance on the multilingual named entity recognition task by 0.3%--0.7% F1, while we achieve up to 2.5% improvement in accuracy on the visual sense disambiguation task. All our code and data are available in: urlhttps://github.com/iacercalixto/visualsem-kg.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

34 of 48 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2017Graph Attention Networkscited by 8,923
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2017Inductive Representation Learning on Large Graphscited by 6,320
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
22 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 932
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  12. 2014Neural Turing Machinescited by 644
  13. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  14. 2014Memory Networkscited by 363
  15. 2019TuckER: Tensor Factorization for Knowledge Graph Completioncited by 290
  16. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  17. 2020How Context Affects Language Models' Factual Predictionscited by 181
  18. 2016Dynamic Memory Networks for Visual and Textual Question Answeringcited by 62
  19. 2020VisualSem: A High-quality Knowledge Graph for Vision and Languagecited by 30
  20. 2016Multimodal Memory Modelling for Video Captioningcited by 20
  21. 2018Learning Visual Knowledge Memory Networks for Visual Question Answeringcited by 10
  22. 2019Cross-lingual Visual Verb Sense Disambiguationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 4; 2025: 1; 2026: 1.

11 papers.

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  2. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  3. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  4. 2022Weisfeiler and Leman Go Relationalcited by 28
  5. 2023A Multi-Task Perspective for Link Prediction with New Relation Types and Nodescited by 24
  6. 2024Noise-powered Multi-modal Knowledge Graph Representation Frameworkcited by 7
  7. 2024Fusing Domain-Specific Content from Large Language Models into Knowledge Graphs for Enhanced Zero Shot Object State Classificationcited by 3
  8. 2024Zero-Shot Relational Learning for Multimodal Knowledge Graphscited by 3
  9. 2022A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Taskscited by 1
  10. 2025Towards Multimodal Graph Large Language Modelcited by 0
  11. 2026LLMs+Graphs: Toward Graph-Native, Synergistic AI Systemscited by 0