# KILM: Knowledge Injection into Encoder-Decoder Language Models

arXiv: https://arxiv.org/abs/2302.09170 · HTML: https://mlsanity.com/paper/2302.09170/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 78 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 30 of 53)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [CM3: A Causal Masked Multimodal Model of the Internet](https://mlsanity.com/paper/2201.07520/) (2022)
- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022)

## Built on by (most cited first, 19 of 19)

- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning](https://mlsanity.com/paper/2404.00213/) (2024)
- [How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances](https://mlsanity.com/paper/2310.07343/) (2023)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [Propagating Knowledge Updates to LMs Through Distillation](https://mlsanity.com/paper/2306.09306/) (2023)
- [Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching](https://mlsanity.com/paper/2406.06326/) (2024)
- [Language Modeling with Editable External Knowledge](https://mlsanity.com/paper/2406.11830/) (2024)
- [Increasing Coverage and Precision of Textual Information in Multilingual Knowledge Graphs](https://mlsanity.com/paper/2311.15781/) (2023)
- [Structure-aware Domain Knowledge Injection for Large Language Models](https://mlsanity.com/paper/2407.16724/) (2024)
- [STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBack](https://mlsanity.com/paper/2410.10584/) (2024)
- [Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models](https://mlsanity.com/paper/2410.09629/) (2024)
- [K-COMP: Retrieval-Augmented Medical Domain Question Answering With Knowledge-Injected Compressor](https://mlsanity.com/paper/2501.13567/) (2025)
- [PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems](https://mlsanity.com/paper/2309.10413/) (2023)
- [ACCORD: Closing the Commonsense Measurability Gap](https://mlsanity.com/paper/2406.02804/) (2024)
- [CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search](https://mlsanity.com/paper/2412.01269/) (2024)
- [Advantages of Domain Knowledge Injection for Legal Document Summarization: A Case Study on Summarizing Indian Court Judgments in English and Hindi](https://mlsanity.com/paper/2602.07382/) (2026)
- [Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting](https://mlsanity.com/paper/2505.00029/) (2025)
- [From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection](https://mlsanity.com/paper/2608.25243/) (2026)
- [Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning](https://mlsanity.com/paper/2608.30987/) (2026)

## Abstract (arXiv)

Large pre-trained language models (PLMs) have been shown to retain implicit knowledge within their parameters. To enhance this implicit knowledge, we propose Knowledge Injection into Language Models (KILM), a novel approach that injects entity-related knowledge into encoder-decoder PLMs, via a generative knowledge infilling objective through continued pre-training. This is done without architectural modifications to the PLMs or adding additional parameters. Experimental results over a suite of knowledge-intensive tasks spanning numerous datasets show that KILM enables models to retain more knowledge and hallucinate less, while preserving their original performance on general NLU and NLG tasks. KILM also demonstrates improved zero-shot performances on tasks such as entity disambiguation, outperforming state-of-the-art models having 30x more parameters.