mlsanityData to 7 Oct 2026

Entity Cloze By Date: What LMs Know About Unseen Entities

arXiv 2205.02832 ↗First posted 5 May 2022cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. However, in a dynamic world, new entities constantly arise. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity. We evaluate LMs' perplexity on masked spans within these sentences. We show that models more informed about the entities, such as those with access to a textual definition of them, achieve lower perplexity on this benchmark. Our experimental results demonstrate that making inferences about new entities remains difficult for LMs. Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge. Our automatic data collection pipeline can be easily used to continually update our benchmark.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 5 May 2022.

Builds on

21 of 37 references.

References · 21

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  8. 2021Editing Factual Knowledge in Language Modelscited by 548
  9. 2021Fast Model Editing at Scalecited by 491
  10. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  11. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  12. 2020Modifying Memories in Transformer Modelscited by 222
9 more
  1. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  2. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  3. 2020Editable Neural Networkscited by 154
  4. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  5. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  6. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  7. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  8. 2014Temporal Analysis of Language through Neural Language Modelscited by 53
  9. 2017Dynamic Word Embeddingscited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 26; 2025: 13; 2026: 3.

55 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  3. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  4. 2022RealTime QA: What's the Answer Right Now?cited by 239
  5. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  6. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  7. 2022Large Language Models with Controllable Working Memorycited by 169
  8. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  9. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  10. 2024Tuning Language Models by Proxycited by 88
  11. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  12. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
38 more
  1. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  2. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  3. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
  4. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  5. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  6. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  7. 2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
  8. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
  9. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  10. 2025SurveyX: Academic Survey Automation via Large Language Modelscited by 27
  11. 2024Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Taskscited by 24
  12. 2024Adaptive Retrieval-Augmented Generation for Conversational Systemscited by 22
  13. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  14. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  15. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
  16. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13
  17. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  18. 2024BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Modelscited by 11
  19. 2024ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Taskscited by 11
  20. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 10
  21. 2024Language Modeling with Editable External Knowledgecited by 9
  22. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  23. 2025Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledgecited by 8
  24. 2024Desiderata for the Context Use of Question Answering Systemscited by 7
  25. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 7
  26. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  27. 2024Into the Unknown: Self-Learning Large Language Modelscited by 5
  28. 2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domaincited by 5
  29. 2024Turkronicles: Diachronic Resources for the Fast Evolving Turkish Languagecited by 3
  30. 2024HALO: Hallucination Analysis and Learning Optimization to Empower LLMs with Retrieval-Augmented Context for Guided Clinical Decision Makingcited by 3
  31. 2024Gradient Localization Improves Lifelong Pretraining of Language Modelscited by 3
  32. 2025PropMEND: Hypernetworks for Knowledge Propagation in LLMscited by 2
  33. 2025HIDE and Seek: Detecting Hallucinations in Language Models via Decoupled Representationscited by 2
  34. 2025Principled Detection of Hallucinations in Large Language Models via Multiple Testingcited by 2
  35. 2025Learning Auxiliary Tasks Improves Reference-Free Hallucination Detection in Open-Domain Long-Form Generationcited by 1
  36. 2024Bag of Lies: Robustness in Continuous Pre-training BERTcited by 0
  37. 2024Detecting Turkish Synonyms Used in Different Time Periodscited by 0
  38. 2025Can Gradient Descent Simulate Prompting?cited by 0

50 most cited of 55 All 55 (JSON)