# Kformer: Knowledge Injection in Transformer Feed-Forward Layers

arXiv: https://arxiv.org/abs/2201.05742 · HTML: https://mlsanity.com/paper/2201.05742/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 15 Jan 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction](https://mlsanity.com/paper/2104.07650/) (2021)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)
- [Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering](https://mlsanity.com/paper/1909.05311/) (2019)
- [Improving Natural Language Inference Using External Knowledge in the Science Questions Domain](https://mlsanity.com/paper/1809.05724/) (2018)
- [Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering](https://mlsanity.com/paper/2005.00691/) (2020)
- [Rethinking the Value of Transformer Components](https://mlsanity.com/paper/2011.03803/) (2020)
- [Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks](https://mlsanity.com/paper/2105.05457/) (2021)
- [How Additional Knowledge can Improve Natural Language Commonsense Question Answering?](https://mlsanity.com/paper/1909.08855/) (2019)

## Built on by (most cited first, 30 of 37)

- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Editing Large Language Models: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2305.13172/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Calibrating Factual Knowledge in Pretrained Language Models](https://mlsanity.com/paper/2210.03329/) (2022)
- [A Survey of Knowledge Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2211.05994/) (2022)
- [OntoProtein: Protein Pretraining With Gene Ontology Embedding](https://mlsanity.com/paper/2201.11147/) (2022)
- [A Study on ReLU and Softmax in Transformer](https://mlsanity.com/paper/2302.06461/) (2023)
- [An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2210.16773/) (2022)
- [Personalized LLM Response Generation with Parameterized Memory Injection](https://mlsanity.com/paper/2404.03565/) (2024)
- [Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearning](https://mlsanity.com/paper/2405.07288/) (2024)
- [Editing Common Sense in Transformers](https://mlsanity.com/paper/2305.14956/) (2023)
- [Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment](https://mlsanity.com/paper/2310.08372/) (2023)
- [RECKONING: Reasoning through Dynamic Knowledge Encoding](https://mlsanity.com/paper/2305.06349/) (2023)
- [Neural Knowledge Bank for Pretrained Transformers](https://mlsanity.com/paper/2208.00399/) (2022)
- [Parameter-Efficient Tuning Makes a Good Classification Head](https://mlsanity.com/paper/2210.16771/) (2022)
- [Potemkin Understanding in Large Language Models](https://mlsanity.com/paper/2506.21521/) (2025)
- [Pretraining with hierarchical memories: separating long-tail and common knowledge](https://mlsanity.com/paper/2510.02375/) (2025)
- [Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks](https://mlsanity.com/paper/2310.15552/) (2023)
- [Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answering](https://mlsanity.com/paper/2410.01401/) (2024)
- ["Merge Conflicts!" Exploring the Impacts of External Distractors to Parametric Knowledge Graphs](https://mlsanity.com/paper/2309.08594/) (2023)
- [Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment](https://mlsanity.com/paper/2310.06365/) (2023)
- [Awakening Augmented Generation: Learning to Awaken Internal Knowledge of Large Language Models for Question Answering](https://mlsanity.com/paper/2403.15268/) (2024)
- [Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedback](https://mlsanity.com/paper/2405.19686/) (2024)
- [Collaborative Knowledge Infusion for Low-resource Stance Detection](https://mlsanity.com/paper/2403.19219/) (2024)
- [Decouple knowledge from parameters for plug-and-play language modeling](https://mlsanity.com/paper/2305.11564/) (2023)
- [FL-Tuning: Layer Tuning for Feed-Forward Network in Transformer](https://mlsanity.com/paper/2206.15312/) (2022)

## Abstract (arXiv)

Recent days have witnessed a diverse set of knowledge injection models for pre-trained language models (PTMs); however, most previous studies neglect the PTMs' own ability with quantities of implicit knowledge stored in parameters. A recent study has observed knowledge neurons in the Feed Forward Network (FFN), which are responsible for expressing factual knowledge. In this work, we propose a simple model, Kformer, which takes advantage of the knowledge stored in PTMs and external knowledge via knowledge injection in Transformer FFN layers. Empirically results on two knowledge-intensive tasks, commonsense reasoning (i.e., SocialIQA) and medical question answering (i.e., MedQA-USMLE), demonstrate that Kformer can yield better performance than other knowledge injection technologies such as concatenation or attention-based injection. We think the proposed simple model and empirical findings may be helpful for the community to develop more powerful knowledge injection methods. Code available in https://github.com/zjunlp/Kformer.