# E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

arXiv: https://arxiv.org/abs/2009.02835 · HTML: https://mlsanity.com/paper/2009.02835/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce](https://mlsanity.com/paper/2308.06966/) (2023)
- [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021)
- [Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey](https://mlsanity.com/paper/2110.08455/) (2021)
- [SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining](https://mlsanity.com/paper/2108.08983/) (2021)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce](https://mlsanity.com/paper/2207.00208/) (2022)
- [LLaMA-E: Empowering E-commerce Authoring with Object-Interleaved Instruction Following](https://mlsanity.com/paper/2308.04913/) (2023)
- [Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding](https://mlsanity.com/paper/2210.03915/) (2022)
- [Learning variant product relationship and variation attributes from e-commerce website structures](https://mlsanity.com/paper/2410.02779/) (2024)
- [Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them](https://mlsanity.com/paper/2503.22006/) (2025)
- [A Survey on Knowledge-Enhanced Pre-trained Language Models](https://mlsanity.com/paper/2212.13428/) (2022)
- [Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue](https://mlsanity.com/paper/2306.03652/) (2023)
- [K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering](https://mlsanity.com/paper/2109.10547/) (2021)
- [SPM: Structured Pretraining and Matching Architectures for Relevance Modeling in Meituan Search](https://mlsanity.com/paper/2308.07711/) (2023)
- [Intelligent Classification and Personalized Recommendation of E-commerce Products Based on Machine Learning](https://mlsanity.com/paper/2403.19345/) (2024)
- [Enhancing E-Commerce Recommendation using Pre-Trained Language Model and Fine-Tuning](https://mlsanity.com/paper/2302.04443/) (2023)
- [An Exploratory Study on Utilising the Web of Linked Data for Product Data Mining](https://mlsanity.com/paper/2109.01411/) (2021)
- [RexBERT: Context Specialized Bidirectional Encoders for E-commerce](https://mlsanity.com/paper/2602.04605/) (2026)

## Abstract (arXiv)

Pre-trained language models such as BERT have achieved great success in a broad range of natural language processing tasks. However, BERT cannot well support E-commerce related tasks due to the lack of two levels of domain knowledge, i.e., phrase-level and product-level. On one hand, many E-commerce tasks require an accurate understanding of domain phrases, whereas such fine-grained phrase-level knowledge is not explicitly modeled by BERT's training objective. On the other hand, product-level knowledge like product associations can enhance the language modeling of E-commerce, but they are not factual knowledge thus using them indiscriminately may introduce noise. To tackle the problem, we propose a unified pre-training framework, namely, E-BERT. Specifically, to preserve phrase-level knowledge, we introduce Adaptive Hybrid Masking, which allows the model to adaptively switch from learning preliminary word knowledge to learning complex phrases, based on the fitting progress of two modes. To utilize product-level knowledge, we introduce Neighbor Product Reconstruction, which trains E-BERT to predict a product's associated neighbors with a denoising cross attention layer. Our investigation reveals promising results in four downstream tasks, i.e., review-based question answering, aspect extraction, aspect sentiment classification, and product classification.