# Pre-Training with Whole Word Masking for Chinese BERT

arXiv: https://arxiv.org/abs/1906.08101 · HTML: https://mlsanity.com/paper/1906.08101/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 14 of 14)

- [Can Authorship Representation Learning Capture Stylistic Features?](https://mlsanity.com/paper/2308.11490/) (2023)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation](https://mlsanity.com/paper/2109.01048/) (2021)
- [QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search](https://mlsanity.com/paper/2306.06707/) (2023)
- [ATHENA: Mathematical Reasoning with Thought Expansion](https://mlsanity.com/paper/2311.01036/) (2023)
- [Attention-aware semantic relevance predicting Chinese sentence reading](https://mlsanity.com/paper/2403.18542/) (2024)
- [RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2412.13008/) (2024)
- [Rank-Aware Negative Training for Semi-Supervised Text Classification](https://mlsanity.com/paper/2306.07621/) (2023)
- [XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad](https://mlsanity.com/paper/2601.14063/) (2026)
- [Quality over Quantity: An Effective Large-Scale Data Reduction Strategy Based on Pointwise V-Information](https://mlsanity.com/paper/2507.00038/) (2025)
- [Multimodal Fine-grained Reasoning for Post Quality Evaluation](https://mlsanity.com/paper/2507.17934/) (2025)
- [Leveraging Human Reading Behavior for Keyphrase Extraction: A Webcam-based Eye-tracking Corpus](https://mlsanity.com/paper/2608.10688/) (2026)
- [A Survey on Fake Review Detection: From Pre-trained Language Models to Large Language Models](https://mlsanity.com/paper/2609.30292/) (2026)
- [Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies](https://mlsanity.com/paper/2610.08879/) (2026)

## Abstract (arXiv)

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we aim to first introduce the whole word masking (wwm) strategy for Chinese BERT, along with a series of Chinese pre-trained language models. Then we also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways. Especially, we propose a new masking strategy called MLM as correction (Mac). To demonstrate the effectiveness of these models, we create a series of Chinese pre-trained language models as our baselines, including BERT, RoBERTa, ELECTRA, RBT, etc. We carried out extensive experiments on ten Chinese NLP tasks to evaluate the created Chinese pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. We open-source our pre-trained language models for further facilitating our research community. Resources are available: https://github.com/ymcui/Chinese-BERT-wwm