Data to 7 Oct 2026

JAKET: Joint Pre-training of Knowledge Graph and Language Understanding

arXiv 2010.00796 ↗First posted 2 Oct 2020cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

Knowledge graphs (KGs) contain rich information about world knowledge, entities and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains a challenge to efficiently integrate information from KG into language modeling. And the understanding of a knowledge graph requires related context. We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and relations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experimental results on several knowledge-aware NLP tasks show that our proposed framework achieves superior performance by effectively leveraging knowledge in language understanding.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 20; 2023: 21; 2024: 14; 2025: 3; 2026: 2.

73 papers.

Sort:

By area:

2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
2023Graph Neural Prompting with Large Language Modelscited by 64
2021JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphscited by 58
2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
2022Knowledge Graph Embedding: A Survey from the Perspective of Representation Spacescited by 41
2021Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attentioncited by 40
2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
2024Knowledge Graph Large Language Model (KG-LLM) for Link Predictioncited by 36

38 more

2022Contrastive Language-Image Pre-Training with Knowledge Graphscited by 34
2023Graph Reasoning for Question Answering with Triplet Retrievalcited by 34
2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
2023Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applicationscited by 33
2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 27
2023KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graphcited by 22
2022Graph Neural Networks Designed for Different Graph Types: A Surveycited by 21
2021KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphscited by 19
2021Knowledge Perceived Multi-modal Pretraining in E-commercecited by 18
2022Visually-Augmented Language Modelingcited by 18
2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
2021Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERTcited by 14
2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14
2021Billion-scale Pre-trained E-commerce Product Knowledge Graph Modelcited by 13
2022Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddingscited by 12
2023Large Knowledge Model: Perspectives and Challengescited by 12
2021Static Embeddings as Efficient Knowledge Bases?cited by 11
2021Structure Inducing Pre-Trainingcited by 10
2022Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfercited by 10
2022KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understandingcited by 9
2022Knowledge Prompting in Pre-trained Language Model for Natural Language Understandingcited by 9
2023Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networkscited by 9
2022A Review of Knowledge Graph Completioncited by 8
2023Entity-Agnostic Representation Learning for Parameter-Efficient Knowledge Graph Embeddingcited by 8
2024ChatEL: Entity Linking with Chatbotscited by 8
2024Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Taskscited by 8
2021Does Knowledge Help General NLU? An Empirical Studycited by 7
2023Structural Information Guided Multimodal Pre-training for Vehicle-centric Perceptioncited by 7
2022KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Modelscited by 6
2023GETT-QA: Graph Embedding based T2T Transformer for Knowledge Graph Question Answeringcited by 6
2023Pre-training Language Models for Comparative Reasoningcited by 6
2023Knowledge Graphs and Pre-trained Language Models enhanced Representation Learning for Conversational Recommender Systemscited by 6
2024EntGPT: Entity Linking with Generative Large Language Modelscited by 6
2024Generative Subgraph Retrieval for Knowledge Graph-Grounded Dialog Generationcited by 6
2023ArK: Augmented Reality with Knowledge Interactive Emergent Abilitycited by 5
2023Knowledge Graphs Queryingcited by 5

50 most cited of 73 All 73 (JSON)