mlsanityData to 7 Oct 2026

Knowledge-Aware Language Model Pretraining

arXiv 2007.00655 ↗First posted 29 Jun 2020cs.CL cs.LG stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

How much knowledge do pretrained language models hold? Recent research observed that pretrained transformers are adept at modeling semantics but it is unclear to what degree they grasp human knowledge, or how to ensure they do so. In this paper we incorporate knowledge-awareness in language model pretraining without changing the transformer architecture, inserting explicit knowledge layers, or adding external storage of semantic information. Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with an entity-extended tokenizer; and at the output, with an additional entity prediction task. Our experiments show that solely by adding these entity signals in pretraining, significantly more knowledge is packed into the transformer parameters: we observe improved language modeling accuracy, factual correctness in LAMA knowledge probing tasks, and semantics in the hidden representations through edge probing.We also show that our knowledge-aware language model (KALM) can serve as a drop-in replacement for GPT-2 models, significantly improving downstream tasks like zero-shot question-answering with no task-related training.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 11; 2023: 12; 2024: 13; 2025: 3; 2026: 1.

52 papers.

Sort:

By area:

  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  5. 2022A Review on Language Models as Knowledge Basescited by 156
  6. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  7. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  8. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  9. 2023Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Taskscited by 73
  10. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  11. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  12. 2022Knowledge Graph Embedding: A Survey from the Perspective of Representation Spacescited by 41
38 more
  1. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  2. 2023Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applicationscited by 33
  3. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  4. 2021FlipDA: Effective and Robust Data Augmentation for Few-Shot Learningcited by 32
  5. 2021Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representationcited by 32
  6. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  7. 2024Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Modelscited by 27
  8. 2021Combining pre-trained language models and structured knowledgecited by 25
  9. 2023KILM: Knowledge Injection into Encoder-Decoder Language Modelscited by 19
  10. 2024Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Actioncited by 19
  11. 2021Knowledge Perceived Multi-modal Pretraining in E-commercecited by 18
  12. 2021XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledgecited by 18
  13. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  14. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14
  15. 2021Billion-scale Pre-trained E-commerce Product Knowledge Graph Modelcited by 13
  16. 2025Sample Efficient Demonstration Selection for In-Context Learningcited by 12
  17. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  18. 2022Knowledge Mining with Scene Text for Fine-Grained Recognitioncited by 10
  19. 2024LLM-Powered Explanations: Unraveling Recommendations Through Subgraph Reasoningcited by 10
  20. 2024CuriousLLM: Elevating Multi-Document Question Answering with LLM-Enhanced Knowledge Graph Reasoningcited by 7
  21. 2024A Theory for Token-Level Harmonization in Retrieval-Augmented Generationcited by 7
  22. 2024Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedbackcited by 6
  23. 2023ArK: Augmented Reality with Knowledge Interactive Emergent Abilitycited by 5
  24. 2023KGLens: Towards Efficient and Effective Knowledge Probing of Large Language Models with Knowledge Graphscited by 5
  25. 2024Knowledge-Centric Templatic Views of Documentscited by 5
  26. 2024EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoningcited by 5
  27. 2022OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialoguecited by 4
  28. 2024DEXTER: A Benchmark for open-domain Complex Question Answering using LLMscited by 4
  29. 2022Does Entity Abstraction Help Generative Transformers Reason?cited by 3
  30. 2022A Survey on Knowledge-Enhanced Pre-trained Language Modelscited by 3
  31. 2023In-Context Ability Transfer for Question Decomposition in Complex QAcited by 3
  32. 2023Furthest Reasoning with Plan Assessment: Stable Reasoning Path with Retrieval-Augmented Large Language Modelscited by 2
  33. 2025BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignmentcited by 2
  34. 2021K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretrainingcited by 1
  35. 2021Linking-Enhanced Pre-Training for Table Semantic Parsingcited by 1
  36. 2023Exploring Category Structure with Contextual Language Models and Lexical Semantic Networkscited by 1
  37. 2022PKGM: A Pre-trained Knowledge Graph Model for E-commerce Applicationcited by 0
  38. 2022ComDensE : Combined Dense Embedding of Relation-aware and Common Features for Knowledge Graph Completioncited by 0

50 most cited of 52 All 52 (JSON)