Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations
Not on the map. Why?
Abstract (arXiv)
This paper explores the robustness of language models (LMs) to variations in the temporal context within factual knowledge. It examines whether LMs can correctly associate a temporal context with a past fact valid over a defined period, by asking them to differentiate correct from incorrect contexts. The LMs' ability to distinguish is analyzed along two dimensions: the distance of the incorrect context from the validity period and the granularity of the context. To this end, a dataset called TimeStress is introduced, enabling the evaluation of 18 diverse LMs. Results reveal that the best LM achieves a perfect distinction for only 11% of the studied facts, with errors, certainly rare, but critical that humans would not make. This work highlights the limitations of current LMs in temporal representation.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 59 references that are in the map. First posted on arXiv 3 Feb 2025.
Builds on
31 of 59 references.
References · 31
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2019Language Models as Knowledge Bases?cited by 1,922
- 2019How Can We Know What Language Models Know?cited by 953
- 2023In-Context Retrieval-Augmented Language Modelscited by 694
- 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
- 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
- 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
- 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
- 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
19 more
- 2022RealTime QA: What's the Answer Right Now?cited by 239
- 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
- 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
- 2021A Dataset for Answering Time-Sensitive Questionscited by 150
- 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
- 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
- 2024Large Language Models Can Learn Temporal Reasoningcited by 110
- 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
- 2023MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRAcited by 80
- 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
- 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
- 2020Temporal Knowledge Base Completion: New Algorithms and Evaluation Protocolscited by 30
- 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
- 2024DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMscited by 23
- 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
- 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 17
- 2024Time Sensitive Knowledge Editing through Efficient Finetuningcited by 14
- 2024BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Modelscited by 11
- 2024WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Modelscited by 9
Built on by
Citing papers per year: 2025: 1.
1 paper.
- 2025Time to Revist Exact Matchcited by 0
- 2025Time to Revist Exact Match20 Sep 2025