mlsanityData to 7 Oct 2026

Entities as Experts: Sparse Memory Access with Entity Supervision

arXiv 2004.07202 ↗First posted 15 Apr 2020cs.CL cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

We focus on the problem of capturing declarative knowledge about entities in the learned parameters of a language model. We introduce a new model - Entities as Experts (EAE) - that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EAE's entity representations are learned directly from text. We show that EAE's learned representations capture sufficient knowledge to answer TriviaQA questions such as "Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?", outperforming an encoder-generator Transformer model with 10x the parameters. According to the LAMA knowledge probes, EAE contains more factual knowledge than a similarly sized BERT, as well as previous approaches that integrate external sources of entity knowledge. Because EAE associates parameters with specific entities, it only needs to access a fraction of its parameters at inference time, and we show that the correct identification and representation of entities is essential to EAE's performance.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 24; 2022: 28; 2023: 16; 2024: 16; 2025: 11; 2026: 4.

100 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  3. 2021Editing Factual Knowledge in Language Modelscited by 548
  4. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  5. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  6. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  7. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  8. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  9. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  10. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  11. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  12. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
38 more
  1. 2022A Review on Language Models as Knowledge Basescited by 156
  2. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  3. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  4. 2021Question Answering Over Temporal Knowledge Graphscited by 99
  5. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  6. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  7. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  8. 2022Retrieval-Enhanced Machine Learningcited by 46
  9. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  10. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  11. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  12. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  13. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  14. 2022Fine-tuning Image Transformers using Learnable Memorycited by 34
  15. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  16. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  17. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  18. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  19. 2022Knowledge Base Question Answering by Case-based Reasoning over Subgraphscited by 31
  20. 2022SKILL: Structured Knowledge Infusion for Large Language Modelscited by 27
  21. 2021TempoQR: Temporal Question Reasoning over Knowledge Graphscited by 26
  22. 2022Improving Time Sensitivity for Question Answering over Temporal Knowledge Graphscited by 26
  23. 2024Question Calibration and Multi-Hop Modeling for Temporal Question Answeringcited by 25
  24. 2025HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generationcited by 24
  25. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  26. 2021Highly Parallel Autoregressive Entity Linking with Discriminative Correctioncited by 21
  27. 2022KALA: Knowledge-Augmented Language Model Adaptationcited by 21
  28. 2021mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Modelscited by 19
  29. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answeringcited by 19
  30. 2021Reasoning Over Virtual Knowledge Bases With Open Predicate Relationscited by 18
  31. 2021XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledgecited by 18
  32. 2023Instructed Language Models with Retrievers Are Powerful Entity Linkerscited by 18
  33. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  34. 2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14
  35. 2023TempEL: Linking Dynamically Evolving and Newly Emerging Entitiescited by 14
  36. 2024Memorization in deep learning: A surveycited by 14
  37. 2021DOCENT: Learning Self-Supervised Entity Representations from Large Document Collectionscited by 13
  38. 2024Grounding Language Models for Visual Entity Recognitioncited by 12

50 most cited of 100 All 100 (JSON)