mlsanityData to 7 Oct 2026

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle

arXiv 2411.08324 ↗First posted 13 Nov 2024cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Many existing evaluation benchmarks for Large Language Models (LLMs) quickly become outdated due to the emergence of new models and training data. These benchmarks also fall short in assessing how LLM performance changes over time, as they consist of a static set of questions without a temporal dimension. To address these limitations, we propose using future event prediction as a continuous evaluation method to assess LLMs' temporal generalization and forecasting abilities. Our benchmark, Daily Oracle, automatically generates question-answer (QA) pairs from daily news, challenging LLMs to predict "future" event outcomes. Our findings reveal that as pre-training data becomes outdated, LLM performance degrades over time. While Retrieval Augmented Generation (RAG) has the potential to enhance prediction accuracy, the performance degradation pattern persists, highlighting the need for continuous model updates. Code and data are available at https://agenticlearning.ai/daily-oracle.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 13 Nov 2024.

Builds on

41 of 63 references.

References · 41

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2023Mistral 7Bcited by 3,065
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2024Mixtral of Expertscited by 1,785
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
29 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  2. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  3. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  4. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  5. 2022RealTime QA: What's the Answer Right Now?cited by 239
  6. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  7. 2023Continual Pre-training of Language Modelscited by 194
  8. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  9. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  10. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  11. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  12. 2023Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Modelscited by 125
  13. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  14. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  15. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  16. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  17. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  18. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  19. 2022Forecasting Future World Events with Neural Networkscited by 63
  20. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  21. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  22. 2021Temporal Effects on Pre-trained Models for Language Processing Taskscited by 46
  23. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  24. 2020ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Datacited by 43
  25. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
  26. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
  27. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  28. 2024Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understandingcited by 23
  29. 2023AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrievalcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 36
  2. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
  3. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  4. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  5. 2025Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Supportcited by 10
  6. 2025TiMoE: Time-Aware Mixture of Language Expertscited by 2
  7. 2026WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoningcited by 2
  8. 2026Agentic Time Machine as an Infrastructure for Future-Event Forecastingcited by 2
  9. 2025Temporal Generalization: A Reality Checkcited by 0
  10. 2026Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streamscited by 0
  11. 2026QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervalscited by 0
  12. 2026Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetrycited by 0
7 more
  1. 2026Decentralized Aggregation of LLM Predictions via Wagering Mechanismscited by 0
  2. 2026Global Merger-Arbitrage Forecasting with Language Modelscited by 0
  3. 2026LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applicationscited by 0
  4. 2026Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgettingcited by 0
  5. 2026Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecastscited by 0
  6. 2026CruxBench: A Benchmark of Information Discoverycited by 0
  7. 2026Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting
  2. 2025When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inference
  3. 2025TransientTables: Evaluating LLMs' Reasoning on Temporally Evolving Semi-structured Tables
  4. 2025Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs
  5. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
  6. 2026SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation
  7. 2025Enterprise Large Language Model Evaluation Benchmark
  8. 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements