Data to 7 Oct 2026

K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

arXiv 2002.01808 ↗First posted 5 Feb 2020cs.CL cs.LGcited by 318

Not on the map. Why?

Abstract (arXiv)

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K-Adapter, a framework that retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowledge in this work, including (1) factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis indicates that K-Adapter captures versatile knowledge than RoBERTa.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 58; 2022: 82; 2023: 83; 2024: 55; 2025: 27; 2026: 13.

318 papers.

Sort:

By area:

2021Learning to Prompt for Continual Learningcited by 938
2022A Survey on In-context Learningcited by 717
2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
2021Fast Model Editing at Scalecited by 491
2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
2021Pre-Trained Models: Past, Present and Futurecited by 319
2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228

38 more

2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
2023Continual Pre-training of Language Modelscited by 194
2023Knowledge Editing for Large Language Models: A Surveycited by 189
2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
2021Towards Continual Knowledge Learning of Language Modelscited by 172
2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
2021Sustainable Modular Debiasing of Language Modelscited by 98
2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
2023Modular Deep Learningcited by 82
2021An Improved Baseline for Sentence-level Relation Extractioncited by 80
2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 56
2022Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuningcited by 50
2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
2022Generative Knowledge Graph Construction: A Reviewcited by 49
2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 47
2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38

50 most cited of 318 All 318 (JSON)