mlsanityData to 13 Oct 2026

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

arXiv 2107.02137 ↗First posted 5 Jul 2021cs.CLcited by 259

Not on the map. Why?

Abstract (arXiv)

Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).

Key facts. Cited by 259 papers in the map (data through 13 Oct 2026). Builds on 68 of its 102 references that are in the map. First posted on arXiv 5 Jul 2021.

Builds on

68 of 102 references.

References · 68

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  7. 2020Scaling Laws for Neural Language Modelscited by 6,494
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  3. 2018Deep contextualized word representationscited by 2,286
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,171
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,851
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,841
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,793
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,130
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,042
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 914
  13. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 833
  14. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 526
  15. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 371
  16. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  17. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  18. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  19. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  20. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  21. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 264
  22. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 263
  23. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  24. 2018A Simple Method for Commonsense Reasoningcited by 223
  25. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 199
  26. 2018CAIL2018: A Large-Scale Legal Dataset for Judgment Predictioncited by 168
  27. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  28. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  29. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  30. 2021M6: A Chinese Multimodal Pretrainercited by 91
  31. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
  32. 2017End-to-End Neural Ad-hoc Ranking with Kernel Poolingcited by 85
  33. 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
  34. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  35. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  36. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  37. 2019Long and Diverse Text Generation with Planning-based Hierarchical Variational Modelcited by 62
  38. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 20; 2022: 49; 2023: 66; 2024: 80; 2025: 36; 2026: 8.

259 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,559
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,229
  3. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,012
  4. 2021Ethical and social risks of harm from Language Modelscited by 964
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  6. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 666
  7. 2023A Comprehensive Overview of Large Language Modelscited by 644
  8. 2024Large Language Models: A Surveycited by 520
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 461
  10. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  11. 2022Large Language Models Are Reasoning Teacherscited by 321
  12. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 317
38 more
  1. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 300
  2. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  3. 2023Challenges and Applications of Large Language Modelscited by 253
  4. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  5. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  6. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  7. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  8. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  9. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 92
  10. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  11. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  12. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 84
  13. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  14. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  15. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  16. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
  17. 2024Vlogger: Make Your Dream A Vlogcited by 70
  18. 2023Graph Neural Prompting with Large Language Modelscited by 64
  19. 2023BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmarkcited by 60
  20. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  21. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  22. 2024Large Language Model Safety: A Holistic Surveycited by 59
  23. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  24. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  25. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  26. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  27. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  28. 2023Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Taskscited by 51
  29. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  30. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  31. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  32. 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathologycited by 43
  33. 2024CoD, Towards an Interpretable Medical Agent using Chain of Diagnosiscited by 43
  34. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  35. 2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
  36. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  37. 2021PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generationcited by 37
  38. 2023Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environmentcited by 37

50 most cited of 259 All 259 (JSON)