Data to 7 Oct 2026

Knowledge Enhanced Contextual Word Representations

arXiv 1909.04164 ↗First posted 9 Sep 2019cs.CLcited by 294

Not on the map. Why?

Abstract (arXiv)

Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual word representations via a form of word-to-entity attention. In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that combines a small amount of entity linking supervision with a large amount of raw text. After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity typing, and word sense disambiguation. KnowBert's runtime is comparable to BERT's and it scales to large KBs.

Key facts. Cited by 294 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 87; 2022: 87; 2023: 61; 2024: 31; 2025: 15; 2026: 12.

294 papers.

Sort:

By area:

2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
2022A Survey on In-context Learningcited by 717
2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
2021Pre-Trained Models: Past, Present and Futurecited by 319
2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180

38 more

2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
2022A Review on Language Models as Knowledge Basescited by 156
2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2022Pretrained Language Models for Text Generation: A Surveycited by 121
2021Question Answering Over Temporal Knowledge Graphscited by 99
2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
2021An Improved Baseline for Sentence-level Relation Extractioncited by 80
2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
2021Re-TACRED: Addressing Shortcomings of the TACRED Datasetcited by 73
2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
2024Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generationcited by 63
2021JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphscited by 58
2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
2021Complex Knowledge Base Question Answering: A Surveycited by 52
2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
2022ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classificationcited by 45
2021Vision Transformer with Progressive Samplingcited by 44
2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37
2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
2021Learning Rich Representation of Keyphrases from Textcited by 36
2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
2021Relation Classification with Entity Type Restrictioncited by 31
2022Summarization as Indirect Supervision for Relation Extractioncited by 31
2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 30
2021Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Modelscited by 29

50 most cited of 294 All 294 (JSON)