mlsanityData to 7 Oct 2026

Static Embeddings as Efficient Knowledge Bases?

arXiv 2104.07094 ↗First posted 14 Apr 2021cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent research investigates factual knowledge stored in large pretrained language models (PLMs). Instead of structural knowledge base (KB) queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. The good performance on this analysis task has been interpreted as PLMs becoming potential repositories of factual knowledge. In experiments across ten linguistically diverse languages, we study knowledge contained in static embeddings. We show that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs. E.g., static embeddings perform 1.6% points better than BERT while just using 0.3% of energy for training. One important factor in their good comparative performance is that static embeddings are standardly learned for a large vocabulary. In contrast, BERT exploits its more sophisticated, but expensive ability to compose meaningful representations from a much smaller subword vocabulary.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

29 of 34 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018Deep contextualized word representationscited by 2,285
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  11. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  12. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
17 more
  1. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  2. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  3. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  4. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  5. 2019Knowledge Enhanced Contextual Word Representationscited by 294
  6. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  7. 2020How Context Affects Language Models' Factual Predictionscited by 181
  8. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  9. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  10. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  11. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  12. 2019E-BERT: Efficient-Yet-Effective Entity Embeddings for BERTcited by 96
  13. 2020Pre-training via Paraphrasingcited by 92
  14. 2019Inducing Relational Knowledge from BERTcited by 75
  15. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  16. 2020BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QAcited by 36
  17. 2017Corpus-level Fine-grained Entity Typingcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 3; 2023: 2; 2024: 1; 2025: 2.

11 papers.

  1. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  2. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  3. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  4. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  5. 2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22
  6. 2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
  7. 2022Transparency Helps Reveal When Language Models Learn Meaningcited by 8
  8. 2024GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledgecited by 7
  9. 2025Multi-Sense Embeddings for Language Models and Knowledge Distillationcited by 3
  10. 2022Probing Commonsense Knowledge in Pre-trained Language Models with Sense-level Precision and Expanded Vocabularycited by 1
  11. 2025Explaining Black-box Language Models with Knowledge Probing Systems: A Post-hoc Explanation Perspectivecited by 1