mlsanityData to 7 Oct 2026

Transformers as Soft Reasoners over Language

arXiv 2002.05867 ↗First posted 14 Feb 2020cs.CL cs.AIcited by 265

Not on the map. Why?

Cited by 134 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Beginning with McCarthy's Advice Taker (1959), AI has pursued the goal of providing a system with explicit, general knowledge and having the system reason over that knowledge. However, expressing the knowledge in a formal (logical or probabilistic) representation has been a major obstacle to this research. This paper investigates a modern approach to this problem where the facts and rules are provided as natural language sentences, thus bypassing a formal representation. We train transformers to reason (or emulate reasoning) over these sentences using synthetically generated data. Our models, that we call RuleTakers, provide the first empirical demonstration that this kind of soft reasoning over language is learnable, can achieve high (99%) accuracy, and generalizes to test data requiring substantially deeper chaining than seen during training (95%+ scores). We also demonstrate that the models transfer well to two hand-authored rulebases, and to rulebases paraphrased into more natural language. These findings are significant as it suggests a new role for transformers, namely as limited "soft theorem provers" operating over explicit theories in language. This in turn suggests new possibilities for explainability, correctability, and counterfactual reasoning in question-answering.

Key facts. Cited by 265 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 56; 2023: 56; 2024: 37; 2025: 52; 2026: 36.

265 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  4. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  6. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  7. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  8. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  9. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  10. 2022Exploring Length Generalization in Large Language Modelscited by 204
  11. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  12. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
38 more
  1. 2021Thinking Like Transformerscited by 182
  2. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  3. 2022Language models show human-like content effects on reasoning taskscited by 171
  4. 2022A Review on Language Models as Knowledge Basescited by 156
  5. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  6. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  7. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  8. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  9. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 107
  10. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  11. 2022On the Paradox of Learning to Reason from Datacited by 93
  12. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 86
  13. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80
  14. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  15. 2024MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypothesescited by 68
  16. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 67
  17. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  18. 2023Large Language Models can Learn Rulescited by 66
  19. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  20. 2023Natural Language Reasoning, A Surveycited by 61
  21. 2023Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translationcited by 59
  22. 2021BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Beliefcited by 56
  23. 2024A Survey on Deep Learning for Theorem Provingcited by 55
  24. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  25. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 52
  26. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  27. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  28. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-trainingcited by 45
  29. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approachescited by 43
  30. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  31. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  32. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  33. 2022Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvementcited by 37
  34. 2023Revealing the structure of language model capabilitiescited by 35
  35. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  36. 2021Analyzing Commonsense Emergence in Few-shot Knowledge Modelscited by 34
  37. 2022Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoningcited by 32
  38. 2022Language Models as Inductive Reasonerscited by 32

50 most cited of 265 All 265 (JSON)