# PERT: Pre-training BERT with Permuted Language Model

arXiv: https://arxiv.org/abs/2203.06906 · HTML: https://mlsanity.com/paper/2203.06906/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [OCNLI: Original Chinese Natural Language Inference](https://mlsanity.com/paper/2010.05444/) (2020)
- [A Span-Extraction Dataset for Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1810.07366/) (2018)
- [DRCD: a Chinese Machine Reading Comprehension Dataset](https://mlsanity.com/paper/1806.00920/) (2018)

## Built on by (most cited first, 23 of 23)

- [Position Prediction as an Effective Pretraining Strategy](https://mlsanity.com/paper/2207.07611/) (2022)
- [MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition](https://mlsanity.com/paper/2404.17113/) (2024)
- [SLING: Sino Linguistic Evaluation of Large Language Models](https://mlsanity.com/paper/2210.11689/) (2022)
- [LERT: A Linguistically-motivated Pre-trained Language Model](https://mlsanity.com/paper/2211.05344/) (2022)
- [BrainNPT: Pre-training of Transformer networks for brain network classification](https://mlsanity.com/paper/2305.01666/) (2023)
- [Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training](https://mlsanity.com/paper/2210.05287/) (2022)
- [FOKE: A Personalized and Explainable Education Framework Integrating Foundation Models, Knowledge Graphs, and Prompt Engineering](https://mlsanity.com/paper/2405.03734/) (2024)
- [Can LLM Substitute Human Labeling? A Case Study of Fine-grained Chinese Address Entity Recognition Dataset for UAV Delivery](https://mlsanity.com/paper/2403.06097/) (2024)
- [Enhancing Generalization in Chain of Thought Reasoning for Smaller Models](https://mlsanity.com/paper/2501.09804/) (2025)
- [An Empirical Study on the Language Modal in Visual Question Answering](https://mlsanity.com/paper/2305.10143/) (2023)
- [IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning](https://mlsanity.com/paper/2306.15273/) (2023)
- [Knowing Where and What: Unified Word Block Pretraining for Document Understanding](https://mlsanity.com/paper/2207.13979/) (2022)
- [Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji](https://mlsanity.com/paper/2504.02116/) (2025)
- [GAME: Generalized deep learning model towards multimodal data integration for early screening of adolescent mental disorders](https://mlsanity.com/paper/2309.10077/) (2023)
- [QASnowball: An Iterative Bootstrapping Framework for High-Quality Question-Answering Data Generation](https://mlsanity.com/paper/2309.10326/) (2023)
- [A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention](https://mlsanity.com/paper/2407.12825/) (2024)
- [Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manuals](https://mlsanity.com/paper/2306.04187/) (2023)
- [Open-Vocabulary Scene Text Recognition via Pseudo-Image Labeling and Margin Loss](https://mlsanity.com/paper/2403.07518/) (2024)
- [Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training](https://mlsanity.com/paper/2404.05560/) (2024)
- [QLSC: A Query Latent Semantic Calibrator for Robust Extractive Question Answering](https://mlsanity.com/paper/2404.19316/) (2024)
- [CARE: A Clue-guided Assistant for CSRs to Read User Manuals](https://mlsanity.com/paper/2408.03633/) (2024)
- [An Ubuntu-Guided Large Language Model Framework for Cognitive Behavioral Mental Health Dialogue](https://mlsanity.com/paper/2601.06875/) (2026)
- [APPSI-139: A Parallel Corpus of English Application Privacy Policy Summarization and Interpretation](https://mlsanity.com/paper/2604.27550/) (2026)

## Abstract (arXiv)

Pre-trained Language Models (PLMs) have been widely used in various natural language processing (NLP) tasks, owing to their powerful text representations trained on large-scale corpora. In this paper, we propose a new PLM called PERT for natural language understanding (NLU). PERT is an auto-encoding model (like BERT) trained with Permuted Language Model (PerLM). The formulation of the proposed PerLM is straightforward. We permute a proportion of the input text, and the training objective is to predict the position of the original token. Moreover, we also apply whole word masking and N-gram masking to improve the performance of PERT. We carried out extensive experiments on both Chinese and English NLU benchmarks. The experimental results show that PERT can bring improvements over various comparable baselines on some of the tasks, while others are not. These results indicate that developing more diverse pre-training tasks is possible instead of masked language model variants. Several quantitative studies are carried out to better understand PERT, which might help design PLMs in the future. Resources are available: https://github.com/ymcui/PERT