Data to 14 Oct 2026

ERNIE: Enhanced Representation through Knowledge Integration

arXiv 1904.09223 ↗First posted 19 Apr 2019cs.CLcited by 302

Not on the map. Why?

Abstract (arXiv)

We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT, ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words.Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit.Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.

Key facts. Cited by 302 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 83; 2022: 100; 2023: 58; 2024: 33; 2025: 16; 2026: 12.

302 papers.

Sort:

By area:

2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
2021Editing Factual Knowledge in Language Modelscited by 553
2021Pre-Trained Models: Past, Present and Futurecited by 319
2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
2022A Review on Language Models as Knowledge Basescited by 157
2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124

38 more

2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 118
2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 112
2021On the Adversarial Robustness of Vision Transformerscited by 106
2021CAT: Cross Attention in Vision Transformercited by 92
2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
2023BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmarkcited by 60
2022Transferability in Deep Learning: A Surveycited by 57
2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
2021A Survey on Green Deep Learningcited by 52
2022POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detectioncited by 51
2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
2022ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrievalcited by 41
2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
2021EventBERT: A Pre-Trained Model for Event Correlation Reasoningcited by 40
2022Towards Robust Ranker for Text Retrievalcited by 37
2021Lexicon Enhanced Chinese Sequence Labeling Using BERT Adaptercited by 34
2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 34
2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
2021HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizationcited by 32
2021Curriculum learning for language modelingcited by 32
2022A Comprehensive Survey on Trustworthy Recommender Systemscited by 32
2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
2023Interactive Natural Language Processingcited by 31

50 most cited of 302 All 302 (JSON)