mlsanityData to 14 Oct 2026

StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

arXiv 2205.11388 ↗First posted 23 May 2022cs.CL cs.LGcited by 126

Not on the map. Why?

Abstract (arXiv)

Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia. However, our world is dynamic, evolves over time, and our models' knowledge becomes outdated. To study how semi-parametric QA models and their underlying parametric language models (LMs) adapt to evolving knowledge, we construct a new large-scale dataset, StreamingQA, with human written and generated questions asked on a given date, to be answered from 14 years of time-stamped news articles. We evaluate our models quarterly as they read new articles not seen in pre-training. We show that parametric models can be updated without full retraining, while avoiding catastrophic forgetting. For semi-parametric models, adding new articles into the search space allows for rapid adaptation, however, models with an outdated underlying LM under-perform those with a retrained LM. For questions about higher-frequency named entities, parametric updates are particularly beneficial. In our dynamic world, the StreamingQA dataset enables a more realistic evaluation of QA models, and our experiments highlight several promising directions for future research.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

32 of 42 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
20 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  2. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  3. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  4. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  5. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  6. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 200
  7. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  8. 2021A Dataset for Answering Time-Sensitive Questionscited by 154
  9. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  10. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  11. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  12. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  13. 2021Question Answering Over Temporal Knowledge Graphscited by 99
  14. 2021Complex Temporal Question Answering on Knowledge Graphscited by 69
  15. 2020TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questionscited by 65
  16. 2017Dynamic Evaluation of Neural Sequence Modelscited by 55
  17. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  18. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  19. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  20. 2020Drinking from a Firehose: Continual Learning with Web-scale Natural Languagecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 26; 2024: 39; 2025: 27; 2026: 28.

126 papers.

Sort:

By area:

  1. 2022Mass-Editing Memory in a Transformercited by 1,001
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  4. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  5. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  6. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  7. 2022RealTime QA: What's the Answer Right Now?cited by 244
  8. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  9. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  10. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 128
  11. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 111
  12. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
38 more
  1. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 88
  2. 2023Context-faithful Prompting for Large Language Modelscited by 80
  3. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 75
  4. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  5. 2025LLM Agents for Education: Advances and Applicationscited by 59
  6. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 44
  7. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  8. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  9. 2023Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learningcited by 37
  10. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Modelscited by 37
  11. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  12. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  13. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  14. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  15. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 26
  16. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  17. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 26
  18. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 25
  19. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 25
  20. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  21. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  22. 2024Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatabilitycited by 22
  23. 2024Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmarkcited by 22
  24. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  25. 2023A Causal View of Entity Bias in (Large) Language Modelscited by 21
  26. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  27. 2022General Intelligence Requires Rethinking Explorationcited by 20
  28. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  29. 2022NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Researchcited by 18
  30. 2023Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Modelscited by 18
  31. 2023Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuningcited by 18
  32. 2024Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Modelscited by 18
  33. 2024MRAG: A Modular Retrieval Framework for Time-Sensitive Question Answeringcited by 17
  34. 2023MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Modelscited by 15
  35. 2024PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answeringcited by 15
  36. 2025TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretrainingcited by 15
  37. 2024AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guaranteescited by 14
  38. 2025It's High Time: A Survey of Temporal Question Answeringcited by 14

50 most cited of 126 All 126 (JSON)