mlsanityData to 7 Oct 2026

KILM: Knowledge Injection into Encoder-Decoder Language Models

arXiv 2302.09170 ↗First posted 17 Feb 2023cs.CL cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Large pre-trained language models (PLMs) have been shown to retain implicit knowledge within their parameters. To enhance this implicit knowledge, we propose Knowledge Injection into Language Models (KILM), a novel approach that injects entity-related knowledge into encoder-decoder PLMs, via a generative knowledge infilling objective through continued pre-training. This is done without architectural modifications to the PLMs or adding additional parameters. Experimental results over a suite of knowledge-intensive tasks spanning numerous datasets show that KILM enables models to retain more knowledge and hallucinate less, while preserving their original performance on general NLU and NLG tasks. KILM also demonstrates improved zero-shot performances on tasks such as entity disambiguation, outperforming state-of-the-art models having 30x more parameters.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 53 of its 78 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

53 of 78 references.

References · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2019How Can We Know What Language Models Know?cited by 953
38 more
  1. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  2. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  3. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  4. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  5. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  6. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  7. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  8. 2020Autoregressive Entity Retrievalcited by 333
  9. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  10. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  11. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  12. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  13. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  14. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  15. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  16. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  17. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  18. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  19. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  20. 2020KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoningcited by 106
  21. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  22. 2019Scalable Zero-shot Entity Linking with Dense Entity Retrievalcited by 85
  23. 2020Sequential Latent Knowledge Selection for Knowledge-Grounded Dialoguecited by 82
  24. 2019Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modelingcited by 71
  25. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  26. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  27. 2018Improving Entity Linking by Modeling Latent Relations between Mentionscited by 53
  28. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  29. 2020Pre-training Text-to-Text Transformers for Concept-centric Common Sensecited by 42
  30. 2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
  31. 2020Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-trainingcited by 24
  32. 2021ReasonBERT: Pre-trained to Reason with Distant Supervisioncited by 22
  33. 2021A Three-Stage Learning Framework for Low-Resource Knowledge-Grounded Dialogue Generationcited by 19
  34. 2021Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triplescited by 13
  35. 2021Text-based NP Enrichmentcited by 8
  36. 2022LM-CORE: Language Models with Contextually Relevant External Knowledgecited by 8
  37. 2020The ApposCorpus: A new multilingual, multi-domain dataset for factual appositive generationcited by 6
  38. 2021Zero-Shot Controlled Generation with Encoder-Decoder Transformerscited by 5

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 9; 2025: 2; 2026: 3.

19 papers.

Sort:

By area:

  1. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  2. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 56
  3. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  4. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  5. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
  6. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  7. 2024Language Modeling with Editable External Knowledgecited by 9
  8. 2023Increasing Coverage and Precision of Textual Information in Multilingual Knowledge Graphscited by 6
  9. 2024Structure-aware Domain Knowledge Injection for Large Language Modelscited by 6
  10. 2024STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBackcited by 4
  11. 2024Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Modelscited by 3
  12. 2025K-COMP: Retrieval-Augmented Medical Domain Question Answering With Knowledge-Injected Compressorcited by 2
7 more
  1. 2023PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systemscited by 1
  2. 2024ACCORD: Closing the Commonsense Measurability Gapcited by 1
  3. 2024CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Searchcited by 1
  4. 2026Advantages of Domain Knowledge Injection for Legal Document Summarization: A Case Study on Summarizing Indian Court Judgments in English and Hindicited by 1
  5. 2025Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgettingcited by 0
  6. 2026From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injectioncited by 0
  7. 2026Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuningcited by 0