mlsanityData to 7 Oct 2026

A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models

arXiv 2202.13392 ↗First posted 27 Feb 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities. In this paper, we propose to build a simple but effective Pluggable Entity Lookup Table (PELT) on demand by aggregating the entity's output representations of multiple occurrences in the corpora. PELT can be compatibly plugged as inputs to infuse supplemental entity knowledge into PLMs. Compared to previous knowledge-enhanced PLMs, PELT only requires 0.2%-5% pre-computation with capability of acquiring knowledge from out-of-domain corpora for domain adaptation scenario. The experiments on knowledge-related tasks demonstrate that our method, PELT, can flexibly and effectively transfer entity knowledge from related corpora into PLMs with different architectures.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 27 Feb 2022.

Builds on

26 of 35 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016Layer Normalizationcited by 5,579
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  9. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  10. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  11. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  12. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
14 more
  1. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  2. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  3. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  4. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  5. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  6. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  7. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  8. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  9. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  10. 2019FewRel 2.0: Towards More Challenging Few-Shot Relation Classificationcited by 94
  11. 2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
  12. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  13. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  14. 2016Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguationcited by 48

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4.

5 papers.

  1. 2023Evaluating Open-QA Evaluationcited by 35
  2. 2023Plug-and-Play Knowledge Injection for Pre-trained Language Modelscited by 6
  3. 2022LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddingscited by 5
  4. 2023Bridge the Gap between Language models and Tabular Understandingcited by 5
  5. 2023UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Modelscited by 0