mlsanityData to 7 Oct 2026

ERNIE: Enhanced Language Representation with Informative Entities

arXiv 1905.07129 ↗First posted 17 May 2019cs.CLcited by 525

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code of this paper can be obtained from https://github.com/thunlp/ERNIE.

Key facts. Cited by 525 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 125; 2022: 139; 2023: 122; 2024: 87; 2025: 38; 2026: 14.

525 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2022A Survey on In-context Learningcited by 717
  5. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  6. 2021Editing Factual Knowledge in Language Modelscited by 548
  7. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  8. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  9. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  10. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  11. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  12. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
38 more
  1. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  2. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  3. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  4. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  5. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  6. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  7. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  8. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  10. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  11. 2022A Review on Language Models as Knowledge Basescited by 156
  12. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  13. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  14. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  15. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  16. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  17. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 126
  18. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  19. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  20. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  21. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  22. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  23. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  24. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  25. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
  26. 2023On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligencecited by 88
  27. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  28. 2021BERTese: Learning to Speak to BERTcited by 82
  29. 2021An Improved Baseline for Sentence-level Relation Extractioncited by 80
  30. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 77
  31. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  32. 2021Re-TACRED: Addressing Shortcomings of the TACRED Datasetcited by 73
  33. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
  34. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  35. 2023Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofittingcited by 70
  36. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  37. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  38. 2022ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classificationcited by 66

50 most cited of 525 All 525 (JSON)