mlsanityData to 7 Oct 2026

Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

arXiv 1502.05698 ↗First posted 19 Feb 2015cs.AI cs.CL stat.MLcited by 309

Not on the map. Why?

Cited by 98 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent. To measure progress towards that goal, we argue for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. We believe many existing learning systems can currently not solve them, and hence our aim is to classify these tasks into skill sets, so that researchers can identify (and then rectify) the failings of their systems. We also extend and improve the recently introduced Memory Networks model, and show it is able to solve some, but not all, of the tasks.

Key facts. Cited by 309 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 43; 2022: 50; 2023: 63; 2024: 55; 2025: 49; 2026: 49.

309 papers.

Sort:

By area:

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  5. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  6. 2023Scaling Data-Constrained Language Modelscited by 391
  7. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  8. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  9. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  10. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  11. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  12. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
38 more
  1. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  2. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  3. 2021PonderNet: Learning to Pondercited by 156
  4. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  5. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  6. 2025BIG-Bench Extra Hardcited by 123
  7. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  8. 2024Are self-explanations from Large Language Models faithful?cited by 101
  9. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  10. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  11. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  12. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  13. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  14. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  15. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  16. 2021Chess as a Testbed for Language Model State Trackingcited by 69
  17. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 69
  18. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  19. 2023Large Language Models can Learn Rulescited by 66
  20. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 65
  21. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  22. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  23. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  24. 2023Natural Language Reasoning, A Surveycited by 61
  25. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  26. 2024Needle In A Multimodal Haystackcited by 56
  27. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  28. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
  29. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 48
  30. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 46
  31. 2022Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systemscited by 44
  32. 2023Entity Tracking in Language Modelscited by 44
  33. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  34. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  35. 2023What's the Magic Word? A Control Theory of LLM Promptingcited by 41
  36. 2021General-Purpose Question-Answering with Macawcited by 40
  37. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  38. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36

50 most cited of 309 All 309 (JSON)