mlsanityData to 7 Oct 2026

SituatedQA: Incorporating Extra-Linguistic Contexts into QA

arXiv 2109.06157 ↗First posted 13 Sep 2021cs.CLcited by 145

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g., roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/ .

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

39 of 60 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
27 more
  1. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  2. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  3. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  4. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  5. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  8. 2018QuAC : Question Answering in Contextcited by 379
  9. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  10. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  11. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  12. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  13. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  14. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  15. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  16. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  17. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
  18. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  19. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  20. 2021Question Answering Over Temporal Knowledge Graphscited by 99
  21. 2017Semi-supervised sequence tagging with bidirectional language modelscited by 86
  22. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  23. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  24. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  25. 2021Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answeringcited by 41
  26. 2020Temporal Common Sense Acquisition with Minimal Supervisioncited by 38
  27. 2019Fine-Grained Temporal Relation Extractioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 20; 2023: 29; 2024: 48; 2025: 28; 2026: 17.

145 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  3. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  4. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  5. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  6. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  7. 2022RealTime QA: What's the Answer Right Now?cited by 239
  8. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  9. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  10. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  11. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Modelscited by 157
  12. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
38 more
  1. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  2. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  3. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  4. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  8. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  9. 2024What Evidence Do Language Models Find Convincing?cited by 74
  10. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 70
  11. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  12. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  13. 2023Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledgecited by 66
  14. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 65
  15. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  16. 2023Selectively Answering Ambiguous Questionscited by 63
  17. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  18. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  19. 2024Aligning Language Models to Explicitly Handle Ambiguitycited by 55
  20. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  21. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 42
  22. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Modelscited by 37
  23. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  24. 2022A Survey of Knowledge-Intensive NLP with Pre-Trained Language Modelscited by 30
  25. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  26. 2021ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answerscited by 29
  27. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  28. 2022INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactionscited by 27
  29. 2022Task Ambiguity in Humans and Language Modelscited by 27
  30. 2023TempTabQA: Temporal Question Answering for Semi-Structured Tablescited by 27
  31. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  32. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
  33. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertaintycited by 25
  34. 2023Asking Clarification Questions to Handle Ambiguity in Open-Domain QAcited by 24
  35. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  36. 2023RELIC: Investigating Large Language Model Responses using Self-Consistencycited by 23
  37. 2024Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understandingcited by 23
  38. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulationcited by 22

50 most cited of 145 All 145 (JSON)