mlsanityData to 14 Oct 2026

KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

arXiv 1911.06136 ↗First posted 13 Nov 2019cs.CLcited by 356

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagE Representation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M, a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.

Key facts. Cited by 356 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 47; 2022: 79; 2023: 92; 2024: 60; 2025: 46; 2026: 32.

356 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  3. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 388
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  6. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  7. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 238
  8. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  9. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  10. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  11. 2022A Review on Language Models as Knowledge Basescited by 157
  12. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
38 more
  1. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  2. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  3. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  4. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 91
  5. 2023Towards Foundation Models for Knowledge Graph Reasoningcited by 86
  6. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  7. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  8. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 78
  9. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  10. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  11. 2022Sequence-to-Sequence Knowledge Graph Completion and Question Answeringcited by 75
  12. 2021NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphscited by 72
  13. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  14. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  15. 2023Large Language Models can Learn Rulescited by 69
  16. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  17. 2021JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphscited by 58
  18. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  19. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  20. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  21. 2021ProoFVer: Natural Logic Theorem Proving for Fact Verificationcited by 53
  22. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  23. 2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
  24. 2022RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Groundingcited by 47
  25. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  26. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  27. 2022Knowledge Graph Embedding: A Survey from the Perspective of Representation Spacescited by 41
  28. 2022A Decade of Knowledge Graphs in Natural Language Processing: A Surveycited by 40
  29. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  30. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  31. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  32. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37
  33. 2023GeoLM: Empowering Language Models for Geospatially Grounded Language Understandingcited by 37
  34. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  35. 2021Learning Rich Representation of Keyphrases from Textcited by 36
  36. 2022CommunityLM: Probing Partisan Worldviews from Language Modelscited by 36
  37. 2024KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completioncited by 36
  38. 2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35

50 most cited of 356 All 356 (JSON)