# K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce arXiv: https://arxiv.org/abs/2104.06960 ยท HTML: https://mlsanity.com/paper/2104.06960/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 53 references that are in the map. - First posted on arXiv 14 Apr 2021. ## Builds on (most cited first, 30 of 35) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017) - [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019) - [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016) - [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019) - [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019) - [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015) - [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019) - [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019) - [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020) - [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019) - [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019) - [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020) - [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020) - [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016) - [Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model](https://mlsanity.com/paper/1912.09637/) (2019) - [From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots](https://mlsanity.com/paper/1801.01957/) (2018) - [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019) - [Modeling Multi-turn Conversation with Deep Utterance Aggregation](https://mlsanity.com/paper/1806.09102/) (2018) - [SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis](https://mlsanity.com/paper/2005.05635/) (2020) - [The JDDC Corpus: A Large-Scale Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Service](https://mlsanity.com/paper/1911.09969/) (2019) ## Built on by (most cited first, 12 of 12) - [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021) - [Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions](https://mlsanity.com/paper/2305.10435/) (2023) - [MAVE: A Product Dataset for Multi-source Attribute Value Extraction](https://mlsanity.com/paper/2112.08663/) (2021) - [EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce](https://mlsanity.com/paper/2308.06966/) (2023) - [LLaMA-E: Empowering E-commerce Authoring with Object-Interleaved Instruction Following](https://mlsanity.com/paper/2308.04913/) (2023) - [Plug-and-Play Knowledge Injection for Pre-trained Language Models](https://mlsanity.com/paper/2305.17691/) (2023) - [ArK: Augmented Reality with Knowledge Interactive Emergent Ability](https://mlsanity.com/paper/2305.00970/) (2023) - [A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generation](https://mlsanity.com/paper/2402.13587/) (2024) - [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022) - [Boosting Multi-Modal E-commerce Attribute Value Extraction via Unified Learning Scheme and Dynamic Range Minimization](https://mlsanity.com/paper/2207.07278/) (2022) - [Mind the Knowledge Gap: A Survey of Knowledge-enhanced Dialogue Systems](https://mlsanity.com/paper/2212.09252/) (2022) - [A Knowledge Plug-and-Play Test Bed for Open-domain Dialogue Generation](https://mlsanity.com/paper/2403.03496/) (2024) ## Abstract (arXiv) Existing pre-trained language models (PLMs) have demonstrated the effectiveness of self-supervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. In this paper, we propose K-PLUG, a knowledge-injected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks. We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge. Specifically, we propose five knowledge-aware self-supervised pre-training objectives to formulate the learning of domain-specific knowledge, including e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities. K-PLUG achieves new state-of-the-art results on a suite of domain-specific NLP tasks, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue, significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks.