mlsanityData to 14 Oct 2026

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

arXiv 1907.12412 ↗First posted 29 Jul 2019cs.CLcited by 231

Not on the map. Why?

Abstract (arXiv)

Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

Key facts. Cited by 231 papers in the map (data through 14 Oct 2026). First posted on arXiv 29 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 62; 2022: 56; 2023: 43; 2024: 41; 2025: 20; 2026: 9.

231 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  4. 2021Editing Factual Knowledge in Language Modelscited by 553
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  7. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  8. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  9. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  10. 2022Can Foundation Models Wrangle Your Data?cited by 175
  11. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  12. 2024Continual Learning for Large Language Models: A Surveycited by 147
38 more
  1. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  2. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 112
  3. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 108
  4. 2023Text Classification via Large Language Modelscited by 106
  5. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  6. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  7. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  8. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 83
  9. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  10. 2024A Survey of Large Language Models in Finance (FinLLMs)cited by 79
  11. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
  12. 2023Open-Vocabulary Video Anomaly Detectioncited by 57
  13. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  14. 2021Read, Listen, and See: Leveraging Multimodal Information Helps Chinese Spell Checkingcited by 52
  15. 2021PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrievalcited by 50
  16. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  17. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  18. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  19. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 41
  20. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  21. 2021Annotating Columns with Pre-trained Language Modelscited by 37
  22. 2021Lexicon Enhanced Chinese Sequence Labeling Using BERT Adaptercited by 34
  23. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  24. 2024GraphEdit: Large Language Models for Graph Structure Learningcited by 33
  25. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  26. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  27. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  28. 2022Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checkingcited by 31
  29. 2022UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidancecited by 30
  30. 2024SciLitLLM: How to Adapt LLMs for Scientific Literature Understandingcited by 29
  31. 2023Pushing the Limits of ChatGPT on NLP Taskscited by 27
  32. 2021Combining pre-trained language models and structured knowledgecited by 25
  33. 2022ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Mapscited by 25
  34. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  35. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  36. 2021ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoderscited by 22
  37. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  38. 2021Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategiescited by 21

50 most cited of 231 All 231 (JSON)