mlsanityData to 14 Oct 2026

NEZHA: Neural Contextualized Representation for Chinese Language Understanding

arXiv 1909.00204 ↗First posted 31 Aug 2019cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks. The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy, Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti) and natural language inference (XNLI).

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 18; 2022: 7; 2023: 7; 2024: 6; 2025: 0; 2026: 1.

39 papers.

Sort:

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  5. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  6. 2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
  7. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  8. 2021NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Predictioncited by 23
  9. 2021ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoderscited by 22
  10. 2021Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Modelscited by 19
  11. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  12. 2021Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarizationcited by 18
27 more
  1. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16
  2. 2023WYWEB: A NLP Evaluation Benchmark For Classical Chinesecited by 12
  3. 2022Domain-Specific NER via Retrieving Correlated Samplescited by 11
  4. 2021Sub-Character Tokenization for Chinese Pretrained Language Modelscited by 7
  5. 2023HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptationcited by 7
  6. 2021Censorship of Online Encyclopedias: Implications for NLP Modelscited by 5
  7. 2023GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmarkcited by 5
  8. 2022Recall, Expand and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typingcited by 4
  9. 2021JABER and SABER: Junior and Senior Arabic BERtcited by 3
  10. 2021You Can Do Better! If You Elaborate the Reason When Making Predictioncited by 2
  11. 2021Sparse-softmax: A Simpler and Faster Alternative Softmax Transformationcited by 2
  12. 2022Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understandingcited by 2
  13. 2022FL-Tuning: Layer Tuning for Feed-Forward Network in Transformercited by 2
  14. 2022Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labelingcited by 2
  15. 2024FarSSiBERT: A Novel Transformer-based Model for Semantic Similarity Measurement of Persian Social Networks Informal Textscited by 2
  16. 2021Multi-Task Learning based Online Dialogic Instruction Detection with Pre-trained Language Modelscited by 1
  17. 2023Unleashing the power of Neural Collapse for Transferability Estimationcited by 1
  18. 2024Vygotsky Distance: Measure for Benchmark Task Similaritycited by 1
  19. 2024MRC-based Nested Medical NER with Co-prediction and Adaptive Pre-trainingcited by 1
  20. 2024SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysiscited by 1
  21. 2024Acoustic Model Optimization over Multiple Data Sources: Merging and Valuationcited by 1
  22. 2021An Effective System for Multi-format Information Extractioncited by 0
  23. 2022Better Pre-Training by Reducing Representation Confusioncited by 0
  24. 2023Address Matching Based On Hierarchical Informationcited by 0
  25. 2023A Simple and Plug-and-play Method for Unsupervised Sentence Representation Enhancementcited by 0
  26. 2024Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-trainingcited by 0
  27. 2026CNM-BERT: A Drop-In Structural Embedding for Chinese Characters via Ideographic Description Sequencescited by 0