# ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders

arXiv: https://arxiv.org/abs/2105.01279 · HTML: https://mlsanity.com/paper/2105.01279/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [A Span-Extraction Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1810.07366/) (2018)
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020)
- [Neural Arabic Question Answering](https://mlsanity.com/paper/1906.05394/) (2019)
- [NEZHA: Neural Contextualized Representation for Chinese Language Understanding](https://mlsanity.com/paper/1909.00204/) (2019)
- [CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model](https://mlsanity.com/paper/2003.01355/) (2020)
- [1.5 billion words Arabic Corpus](https://mlsanity.com/paper/1611.04033/) (2016)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [Named Entity Recognition for Social Media Texts with Semantic Augmentation](https://mlsanity.com/paper/2010.15458/) (2020)
- [Improving Constituency Parsing with Span Attention](https://mlsanity.com/paper/2010.07543/) (2020)
- [Improving Named Entity Recognition with Attentive Ensemble of Syntactic Information](https://mlsanity.com/paper/2010.15466/) (2020)
- [Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention](https://mlsanity.com/paper/1911.02821/) (2019)

## Built on by (most cited first, 22 of 22)

- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence](https://mlsanity.com/paper/2209.02970/) (2022)
- [RevCore: Review-augmented Conversational Recommendation](https://mlsanity.com/paper/2106.00957/) (2021)
- [ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2112.12731/) (2021)
- [N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution](https://mlsanity.com/paper/2211.11436/) (2022)
- [ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences](https://mlsanity.com/paper/2311.06025/) (2023)
- [Word Graph Guided Summarization for Radiology Findings](https://mlsanity.com/paper/2112.09925/) (2021)
- [Ziya2: Data-centric Learning is All LLMs Need](https://mlsanity.com/paper/2311.03301/) (2023)
- [Improving Image Captioning via Predicting Structured Concepts](https://mlsanity.com/paper/2311.08223/) (2023)
- [Multimodal Conditioned Diffusive Time Series Forecasting](https://mlsanity.com/paper/2504.19669/) (2025)
- [Improving Arabic Diacritization by Learning to Diacritize and Translate](https://mlsanity.com/paper/2109.14150/) (2021)
- [Balanced Training Data Augmentation for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2507.09485/) (2025)
- [Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation](https://mlsanity.com/paper/2602.06359/) (2026)
- [Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text](https://mlsanity.com/paper/2306.03557/) (2023)
- [Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing](https://mlsanity.com/paper/2506.07086/) (2025)
- [Context Enhanced Short Text Matching using Clickthrough Data](https://mlsanity.com/paper/2203.01849/) (2022)
- [Detoxification of Large Language Models through Output-layer Fusion with a Calibration Model](https://mlsanity.com/paper/2506.01266/) (2025)
- [Computed Tomography Visual Question Answering with Cross-modal Feature Graphing](https://mlsanity.com/paper/2507.04333/) (2025)
- [Frustratingly Easy Task-aware Pruning for Large Language Models](https://mlsanity.com/paper/2510.22489/) (2025)
- [ChiMed 2.0: Advancing Chinese Medical Dataset in Facilitating Large Language Modeling](https://mlsanity.com/paper/2507.15275/) (2025)
- [Explainable Multimodal Aspect-Based Sentiment Analysis with Dependency-guided Large Language Model](https://mlsanity.com/paper/2601.06848/) (2026)

## Abstract (arXiv)

Pre-trained text encoders have drawn sustaining attention in natural language processing (NLP) and shown their capability in obtaining promising results in different tasks. Recent studies illustrated that external self-supervised signals (or knowledge extracted by unsupervised learning, such as n-grams) are beneficial to provide useful semantic evidence for understanding languages such as Chinese, so as to improve the performance on various downstream tasks accordingly. To further enhance the encoders, in this paper, we propose to pre-train n-gram-enhanced encoders with a large volume of data and advanced techniques for training. Moreover, we try to extend the encoder to different languages as well as different domains, where it is confirmed that the same architecture is applicable to these varying circumstances and new state-of-the-art performance is observed from a long list of NLP tasks across languages and domains.