Data to 14 Oct 2026

E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

arXiv 1911.03681 ↗First posted 9 Nov 2019cs.CLcited by 96

Not on the map. Why?

Abstract (arXiv)

We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pretraining of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 33; 2022: 22; 2023: 25; 2024: 14; 2025: 2.

96 papers.

Sort:

By area:

2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
2021True Few-Shot Learning with Language Modelscited by 344
2021Pre-Trained Models: Past, Present and Futurecited by 319
2021Towards Continual Knowledge Learning of Language Modelscited by 174
2021Meta-learning via Language Model In-context Tuningcited by 164
2022A Review on Language Models as Knowledge Basescited by 157
2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2022Domain Adaptation via Prompt Learningcited by 128
2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92

38 more

2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
2021Can Language Models be Biomedical Knowledge Bases?cited by 83
2022Understanding Transformer Memorization Recall Through Idiomscited by 62
2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 49
2022How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysiscited by 42
2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
2023SoK: Memorization in General-Purpose Large Language Modelscited by 37
2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22
2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
2023Evaluating Language Models for Knowledge Base Completioncited by 20
2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
2022Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typingcited by 16
2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14
2021Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triplescited by 13
2024Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extractioncited by 13
2021Injecting Knowledge Base Information into End-to-End Joint Entity and Relation Extraction and Coreference Resolutioncited by 12
2023Calibration of Transformer-based Models for Identifying Stress and Depression in Social Mediacited by 12
2021Static Embeddings as Efficient Knowledge Bases?cited by 11
2021Conversational Entity Linking: Problem Definition and Datasetscited by 11
2021How to Query Language Models?cited by 11
2022Detection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problemcited by 11
2022A survey on knowledge-enhanced multimodal learningcited by 11
2023SpEL: Structured Prediction for Entity Linkingcited by 11
2024BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Modelscited by 11
2021Structure Inducing Pre-Trainingcited by 10
2021Using Knowledge-Embedded Attention to Augment Pre-trained Language Models for Fine-Grained Emotion Recognitioncited by 10
2022Entity-aware Transformers for Entity Searchcited by 10
2023On the Evolution of Knowledge Graphs: A Survey and Perspectivecited by 10
2021LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Frameworkcited by 9

50 most cited of 96 All 96 (JSON)