mlsanityData to 14 Oct 2026

Revisiting Pre-Trained Models for Chinese Natural Language Processing

arXiv 2004.13922 ↗First posted 29 Apr 2020cs.CLcited by 266

Not on the map. Why?

Abstract (arXiv)

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we target on revisiting Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pre-trained language model series to the community. We also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac). We carried out extensive experiments on eight Chinese NLP tasks to revisit the existing pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. Resources available: https://github.com/ymcui/MacBERT

Key facts. Cited by 266 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 43; 2022: 67; 2023: 64; 2024: 39; 2025: 26; 2026: 27.

266 papers.

Sort:

By area:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  2. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  3. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  4. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 121
  5. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  6. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 109
  7. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  8. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 101
  9. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  10. 2021M6: A Chinese Multimodal Pretrainercited by 91
  11. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  12. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
38 more
  1. 2023Multiscale Positive-Unlabeled Detection of AI-Generated Textscited by 61
  2. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  3. 2022The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for Chinese Spell Checkingcited by 56
  4. 2023Multilingual Language Models are not Multicultural: A Case Study in Emotioncited by 54
  5. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  6. 2023Huatuo-26M, a Large-scale Chinese Medical QA Datasetcited by 46
  7. 2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
  8. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  9. 2021Neural-Symbolic Solver for Math Word Problems with Auxiliary Taskscited by 38
  10. 2024FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysiscited by 36
  11. 2022MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correctioncited by 33
  12. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  13. 2021MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solvingcited by 32
  14. 2022SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parsercited by 32
  15. 2021More but Correct: Generating Diversified and Entity-revised Medical Responsecited by 31
  16. 2022Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checkingcited by 31
  17. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  18. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  19. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26
  20. 2022Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correctioncited by 25
  21. 2021SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Miningcited by 24
  22. 2021LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generationcited by 24
  23. 2022LogicSolver: Towards Interpretable Math Word Problem Solving with Logical Prompt-enhanced Learningcited by 24
  24. 2022PERT: Pre-training BERT with Permuted Language Modelcited by 23
  25. 2024MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognitioncited by 23
  26. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Modelscited by 23
  27. 2021ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoderscited by 22
  28. 2022Boundary Smoothing for Named Entity Recognitioncited by 22
  29. 2022CCMB: A Large-scale Chinese Cross-modal Benchmarkcited by 21
  30. 2022GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminatorcited by 20
  31. 2023Modeling Entities as Semantic Points for Visual Information Extraction in the Wildcited by 20
  32. 2023Personality Understanding of Fictional Characters during Book Readingcited by 20
  33. 2024Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challengescited by 20
  34. 2021Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Modelscited by 19
  35. 2022Parallel Instance Query Network for Named Entity Recognitioncited by 19
  36. 2022WeLM: A Well-Read Pre-trained Language Model for Chinesecited by 19
  37. 2023CORGI-PM: A Chinese Corpus For Gender Bias Probing and Mitigationcited by 19
  38. 2023Rethinking Masked Language Modeling for Chinese Spelling Correctioncited by 19

50 most cited of 266 All 266 (JSON)