# Injecting Domain Knowledge in Language Models for Task-Oriented Dialogue Systems

arXiv: https://arxiv.org/abs/2212.08120 · HTML: https://mlsanity.com/paper/2212.08120/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [Dialogue State Tracking with a Language Model using Schema-Driven Prompting](https://mlsanity.com/paper/2109.07506/) (2021)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers](https://mlsanity.com/paper/2005.11787/) (2020)
- [Augmenting Transformers with KNN-Based Composite Memory for Dialogue](https://mlsanity.com/paper/2004.12744/) (2020)
- [The Adapter-Bot: All-In-One Controllable Conversational Model](https://mlsanity.com/paper/2008.12579/) (2020)
- [Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2009.13656/) (2020)
- [Combining pre-trained language models and structured knowledge](https://mlsanity.com/paper/2101.12294/) (2021)
- [KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs](https://mlsanity.com/paper/2109.04223/) (2021)

## Built on by (most cited first, 6 of 6)

- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration](https://mlsanity.com/paper/2402.11441/) (2024)
- [What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition](https://mlsanity.com/paper/2401.12756/) (2024)
- [Few-Shot Table-to-Text Generation with Prompt Planning and Knowledge Memorization](https://mlsanity.com/paper/2302.04415/) (2023)
- [Adapting Knowledge for Few-shot Table-to-Text Generation](https://mlsanity.com/paper/2302.12468/) (2023)
- [Adapter-based Approaches to Knowledge-enhanced Language Models -- A Survey](https://mlsanity.com/paper/2411.16403/) (2024)

## Abstract (arXiv)

Pre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data. Previous studies augmented PLMs with symbolic knowledge for different downstream NLP tasks. However, knowledge bases (KBs) utilized in these studies are usually large-scale and static, in contrast to small, domain-specific, and modifiable knowledge bases that are prominent in real-world task-oriented dialogue (TOD) systems. In this paper, we showcase the advantages of injecting domain-specific knowledge prior to fine-tuning on TOD tasks. To this end, we utilize light-weight adapters that can be easily integrated with PLMs and serve as a repository for facts learned from different KBs. To measure the efficacy of proposed knowledge injection methods, we introduce Knowledge Probing using Response Selection (KPRS) -- a probe designed specifically for TOD models. Experiments on KPRS and the response generation task show improvements of knowledge injection with adapters over strong baselines.