# ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding

arXiv: https://arxiv.org/abs/2010.12148 · HTML: https://mlsanity.com/paper/2010.12148/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 17 of 17)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [MelodyGLM: Multi-task Pre-training for Symbolic Melody Generation](https://mlsanity.com/paper/2309.10738/) (2023)
- [Sub-Character Tokenization for Chinese Pretrained Language Models](https://mlsanity.com/paper/2106.00400/) (2021)
- [N-gram Prediction and Word Difference Representations for Language Modeling](https://mlsanity.com/paper/2409.03295/) (2024)
- [Assistive Recipe Editing through Critiquing](https://mlsanity.com/paper/2205.02454/) (2022)
- [CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations](https://mlsanity.com/paper/2208.10844/) (2022)
- [Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling](https://mlsanity.com/paper/2210.15231/) (2022)
- [Prompting is not Enough: Exploring Knowledge Integration and Controllable Generation](https://mlsanity.com/paper/2505.19660/) (2025)
- [Enhancing Rumor Detection Methods with Propagation Structure Infused Language Model](https://mlsanity.com/paper/2508.07209/) (2025)
- [Analysing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasets](https://mlsanity.com/paper/2110.15712/) (2021)
- [Linguistic Knowledge in Data Augmentation for Natural Language Processing: An Example on Chinese Question Matching](https://mlsanity.com/paper/2111.14709/) (2021)
- [Leveraging Knowledge and Reinforcement Learning for Enhanced Reliability of Language Models](https://mlsanity.com/paper/2308.13467/) (2023)
- [Probabilistic Linguistic Knowledge and Token-level Text Augmentation](https://mlsanity.com/paper/2306.16644/) (2023)
- [Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training](https://mlsanity.com/paper/2404.05560/) (2024)
- [N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition](https://mlsanity.com/paper/2603.03930/) (2026)
- [POSPAN: Position-Constrained Span Masking for Language Model Pre-training](https://mlsanity.com/paper/2609.16061/) (2026)

## Abstract (arXiv)

Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT's Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of n tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained information into pre-training. In ERNIE-Gram, n-grams are masked and predicted directly using explicit n-gram identities rather than contiguous sequences of n tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible n-gram identities as optional n-gram masks and predict them in both coarse-grained and fine-grained manners to enable comprehensive n-gram prediction and relation modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art methods. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.