mlsanityData to 14 Oct 2026

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

arXiv 2101.02235 ↗First posted 6 Jan 2021cs.CLcited by 743

Not on the map. Why?

Cited by 289 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of ~66%.

Key facts. Cited by 743 papers in the map (data through 14 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 6 Jan 2021.

Builds on

23 of 30 references.

References · 23

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  10. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  11. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
11 more
  1. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  2. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  3. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  4. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  5. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  6. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  7. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  8. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  9. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  10. 2020Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Modelscited by 65
  11. 2019Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QAcited by 63

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 62; 2023: 161; 2024: 205; 2025: 178; 2026: 125.

743 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2023A Survey of Large Language Modelscited by 2,576
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  9. 2023PaLM 2 Technical Reportcited by 1,162
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  12. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
38 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  3. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2023A Comprehensive Overview of Large Language Modelscited by 648
  7. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  8. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  9. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  10. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  11. 2023Active Retrieval Augmented Generationcited by 516
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  13. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  14. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  15. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  16. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  17. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  18. 2023Language Models can Solve Computer Taskscited by 372
  19. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  20. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  21. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  22. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  23. 2022Large Language Models Are Reasoning Teacherscited by 329
  24. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  25. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  26. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  27. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  28. 2022UL2: Unifying Language Learning Paradigmscited by 295
  29. 2023Faithful Chain-of-Thought Reasoningcited by 288
  30. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  31. 2022Teaching Small Language Models to Reasoncited by 284
  32. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  33. 2023A Survey on Model Compression for Large Language Modelscited by 269
  34. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  35. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  36. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  37. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  38. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252

50 most cited of 743 All 743 (JSON)