mlsanityData to 7 Oct 2026

ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders

arXiv 2105.01279 ↗First posted 4 May 2021cs.CL cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Pre-trained text encoders have drawn sustaining attention in natural language processing (NLP) and shown their capability in obtaining promising results in different tasks. Recent studies illustrated that external self-supervised signals (or knowledge extracted by unsupervised learning, such as n-grams) are beneficial to provide useful semantic evidence for understanding languages such as Chinese, so as to improve the performance on various downstream tasks accordingly. To further enhance the encoders, in this paper, we propose to pre-train n-gram-enhanced encoders with a large volume of data and advanced techniques for training. Moreover, we try to extend the encoder to different languages as well as different domains, where it is confirmed that the same architecture is applicable to these varying circumstances and new state-of-the-art performance is observed from a long list of NLP tasks across languages and domains.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 4 May 2021.

Builds on

23 of 42 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  8. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 359
  9. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  10. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  11. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  12. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
11 more
  1. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  2. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  3. 2019Neural Arabic Question Answeringcited by 48
  4. 2019NEZHA: Neural Contextualized Representation for Chinese Language Understandingcited by 39
  5. 2020CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Modelcited by 36
  6. 20161.5 billion words Arabic Corpuscited by 35
  7. 2019ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representationscited by 35
  8. 2020Named Entity Recognition for Social Media Texts with Semantic Augmentationcited by 23
  9. 2020Improving Constituency Parsing with Span Attentioncited by 17
  10. 2020Improving Named Entity Recognition with Attentive Ensemble of Syntactic Informationcited by 17
  11. 2019Enhancing Pre-trained Chinese Character Representation with Word-aligned Attentioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 4; 2024: 0; 2025: 7; 2026: 2.

22 papers.

Sort:

By area:

  1. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  2. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  3. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  4. 2021RevCore: Review-augmented Conversational Recommendationcited by 47
  5. 2022N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolutioncited by 46
  6. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  7. 2023ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferencescited by 43
  8. 2021Word Graph Guided Summarization for Radiology Findingscited by 19
  9. 2023Ziya2: Data-centric Learning is All LLMs Needcited by 15
  10. 2023Improving Image Captioning via Predicting Structured Conceptscited by 12
  11. 2025Multimodal Conditioned Diffusive Time Series Forecastingcited by 8
  12. 2021Improving Arabic Diacritization by Learning to Diacritize and Translatecited by 6
10 more
  1. 2025Balanced Training Data Augmentation for Aspect-Based Sentiment Analysiscited by 5
  2. 2023Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Textcited by 2
  3. 2025Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computingcited by 2
  4. 2026Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptationcited by 2
  5. 2022Context Enhanced Short Text Matching using Clickthrough Datacited by 1
  6. 2025Detoxification of Large Language Models through Output-layer Fusion with a Calibration Modelcited by 1
  7. 2025Computed Tomography Visual Question Answering with Cross-modal Feature Graphingcited by 1
  8. 2025Frustratingly Easy Task-aware Pruning for Large Language Modelscited by 1
  9. 2025ChiMed 2.0: Advancing Chinese Medical Dataset in Facilitating Large Language Modelingcited by 0
  10. 2026Explainable Multimodal Aspect-Based Sentiment Analysis with Dependency-guided Large Language Modelcited by 0