# K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering

arXiv: https://arxiv.org/abs/2109.10547 · HTML: https://mlsanity.com/paper/2109.10547/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model](https://mlsanity.com/paper/1912.09637/) (2019)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://mlsanity.com/paper/2004.07202/) (2020)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [Automated Phrase Mining from Massive Text Corpora](https://mlsanity.com/paper/1702.04457/) (2017)
- [Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases](https://mlsanity.com/paper/2009.11564/) (2020)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [Simple and Effective Text Matching with Richer Alignment Features](https://mlsanity.com/paper/1908.00300/) (2019)
- [AliMe Assist: An Intelligent Assistant for Creating an Innovative E-commerce Experience](https://mlsanity.com/paper/1801.05032/) (2018)
- [Integrating Graph Contextualized Knowledge into Pre-trained Language Models](https://mlsanity.com/paper/1912.00147/) (2019)
- [OpenHowNet: An Open Sememe-based Lexical Knowledge Base](https://mlsanity.com/paper/1901.09957/) (2019)
- [E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](https://mlsanity.com/paper/2009.02835/) (2020)
- [AliMe KG: Domain Knowledge Graph Construction and Application in E-commerce](https://mlsanity.com/paper/2009.11684/) (2020)

## Built on by (most cited first, 2 of 2)

- [Distinguish Before Answer: Generating Contrastive Explanation as Knowledge for Commonsense Question Answering](https://mlsanity.com/paper/2305.08135/) (2023)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)

## Abstract (arXiv)

Knowledge enhanced pre-trained language models (K-PLMs) are shown to be effective for many public tasks in the literature but few of them have been successfully applied in practice. To address this problem, we propose K-AID, a systematic approach that includes a low-cost knowledge acquisition process for acquiring domain knowledge, an effective knowledge infusion module for improving model performance, and a knowledge distillation component for reducing the model size and deploying K-PLMs on resource-restricted devices (e.g., CPU) for real-world application. Importantly, instead of capturing entity knowledge like the majority of existing K-PLMs, our approach captures relational knowledge, which contributes to better-improving sentence-level text classification and text matching tasks that play a key role in question answering (QA). We conducted a set of experiments on five text classification tasks and three text matching tasks from three domains, namely E-commerce, Government, and Film&TV, and performed online A/B tests in E-commerce. Experimental results show that our approach is able to achieve substantial improvement on sentence-level question answering tasks and bring beneficial business value in industrial settings.