# NEZHA: Neural Contextualized Representation for Chinese Language Understanding

arXiv: https://arxiv.org/abs/1909.00204 · HTML: https://mlsanity.com/paper/1909.00204/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2019.

## Built on by (most cited first, 30 of 39)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://mlsanity.com/paper/2104.12369/) (2021)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://mlsanity.com/paper/2110.06696/) (2021)
- [FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark](https://mlsanity.com/paper/2107.07498/) (2021)
- [NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction](https://mlsanity.com/paper/2109.03564/) (2021)
- [ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders](https://mlsanity.com/paper/2105.01279/) (2021)
- [Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models](https://mlsanity.com/paper/2104.07204/) (2021)
- [Improving Tokenisation by Alternative Treatment of Spaces](https://mlsanity.com/paper/2204.04058/) (2022)
- [Hierarchical Cross-Modality Semantic Correlation Learning Model for Multimodal Summarization](https://mlsanity.com/paper/2112.12072/) (2021)
- [Training Multilingual Pre-trained Language Model with Byte-level Subwords](https://mlsanity.com/paper/2101.09469/) (2021)
- [WYWEB: A NLP Evaluation Benchmark For Classical Chinese](https://mlsanity.com/paper/2305.14150/) (2023)
- [Domain-Specific NER via Retrieving Correlated Samples](https://mlsanity.com/paper/2208.12995/) (2022)
- [Sub-Character Tokenization for Chinese Pretrained Language Models](https://mlsanity.com/paper/2106.00400/) (2021)
- [HiFi: High-Information Attention Heads Hold for Parameter-Efficient Model Adaptation](https://mlsanity.com/paper/2305.04573/) (2023)
- [Censorship of Online Encyclopedias: Implications for NLP Models](https://mlsanity.com/paper/2101.09294/) (2021)
- [GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2305.06545/) (2023)
- [Recall, Expand and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing](https://mlsanity.com/paper/2212.09125/) (2022)
- [JABER and SABER: Junior and Senior Arabic BERt](https://mlsanity.com/paper/2112.04329/) (2021)
- [You Can Do Better! If You Elaborate the Reason When Making Prediction](https://mlsanity.com/paper/2103.14919/) (2021)
- [Sparse-softmax: A Simpler and Faster Alternative Softmax Transformation](https://mlsanity.com/paper/2112.12433/) (2021)
- [Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding](https://mlsanity.com/paper/2205.10687/) (2022)
- [FL-Tuning: Layer Tuning for Feed-Forward Network in Transformer](https://mlsanity.com/paper/2206.15312/) (2022)
- [Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling](https://mlsanity.com/paper/2210.15231/) (2022)
- [FarSSiBERT: A Novel Transformer-based Model for Semantic Similarity Measurement of Persian Social Networks Informal Texts](https://mlsanity.com/paper/2407.19173/) (2024)
- [Multi-Task Learning based Online Dialogic Instruction Detection with Pre-trained Language Models](https://mlsanity.com/paper/2107.07119/) (2021)
- [Unleashing the power of Neural Collapse for Transferability Estimation](https://mlsanity.com/paper/2310.05754/) (2023)
- [Vygotsky Distance: Measure for Benchmark Task Similarity](https://mlsanity.com/paper/2402.14890/) (2024)

## Abstract (arXiv)

The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks. The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy, Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti) and natural language inference (XNLI).