# A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models

arXiv: https://arxiv.org/abs/2202.13392 · HTML: https://mlsanity.com/paper/2202.13392/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 27 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model](https://mlsanity.com/paper/1912.09637/) (2019)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://mlsanity.com/paper/2004.07202/) (2020)
- [E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](https://mlsanity.com/paper/1911.03681/) (2019)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [A Mutual Information Maximization Perspective of Language Representation Learning](https://mlsanity.com/paper/1910.08350/) (2019)
- [Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?](https://mlsanity.com/paper/2106.01561/) (2021)
- [Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation](https://mlsanity.com/paper/1601.01343/) (2016)

## Built on by (most cited first, 5 of 5)

- [Evaluating Open-QA Evaluation](https://mlsanity.com/paper/2305.12421/) (2023)
- [Plug-and-Play Knowledge Injection for Pre-trained Language Models](https://mlsanity.com/paper/2305.17691/) (2023)
- [LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddings](https://mlsanity.com/paper/2210.00305/) (2022)
- [Bridge the Gap between Language models and Tabular Understanding](https://mlsanity.com/paper/2302.09302/) (2023)
- [UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models](https://mlsanity.com/paper/2305.01624/) (2023)

## Abstract (arXiv)

Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities. In this paper, we propose to build a simple but effective Pluggable Entity Lookup Table (PELT) on demand by aggregating the entity's output representations of multiple occurrences in the corpora. PELT can be compatibly plugged as inputs to infuse supplemental entity knowledge into PLMs. Compared to previous knowledge-enhanced PLMs, PELT only requires 0.2%-5% pre-computation with capability of acquiring knowledge from out-of-domain corpora for domain adaptation scenario. The experiments on knowledge-related tasks demonstrate that our method, PELT, can flexibly and effectively transfer entity knowledge from related corpora into PLMs with different architectures.