mlsanityData to 7 Oct 2026

Language models show human-like content effects on reasoning tasks

arXiv 2207.07051 ↗First posted 14 Jul 2022cs.CL cs.AI cs.LGcited by 171

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Reasoning is a key ability for an intelligent system. Large language models (LMs) achieve above-chance performance on abstract reasoning tasks, but exhibit many imperfections. However, human abstract reasoning is also imperfect. For example, human reasoning is affected by our real-world knowledge and beliefs, and shows notable "content effects"; humans reason more reliably when the semantic content of a problem supports the correct logical inferences. These content-entangled reasoning patterns play a central role in debates about the fundamental nature of human intelligence. Here, we investigate whether language models unicodex2014 whose prior expectations capture some aspects of human knowledge unicodex2014 similarly mix content into their answers to logical problems. We explored this question across three logical reasoning tasks: natural language inference, judging the logical validity of syllogisms, and the Wason selection task. We evaluate state of the art large language models, as well as humans, and find that the language models reflect many of the same patterns observed in humans across these tasks unicodex2014 like humans, models answer more accurately when the semantic content of a task supports the logical inferences. These parallels are reflected both in answer patterns, and in lower-level features like the relationship between model answer distributions and human response times. Our findings have implications for understanding both these cognitive effects in humans, and the factors that contribute to language model performance.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 51 of its 113 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

51 of 113 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoning15 Feb
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2019Language Models as Knowledge Bases?cited by 1,922
37 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  9. 2022Using cognitive psychology to understand GPT-3cited by 389
  10. 2022Self-critiquing models for assisting human evaluatorscited by 372
  11. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  12. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  13. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  14. 2020Transformers as Soft Reasoners over Languagecited by 265
  15. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  16. 2022Can language models learn from explanations in context?cited by 248
  17. 2018A Simple Method for Commonsense Reasoningcited by 223
  18. 2022Predictability and Surprise in Large Generative Modelscited by 209
  19. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  20. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  21. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  22. 2017The Consciousness Priorcited by 105
  23. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  24. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  25. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  26. 2017Thinking Fast and Slow: Optimization Decomposition Across Timescalescited by 60
  27. 2017Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Studycited by 46
  28. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  29. 2020Meta-learning in natural and artificial intelligencecited by 39
  30. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  31. 2018Evaluating Compositionality in Sentence Embeddingscited by 35
  32. 2021Symbolic Behaviour in Artificial Intelligencecited by 24
  33. 2021Explanatory models in neuroscience: Part 1 -- taking mechanistic abstraction seriouslycited by 18
  34. 2019Towards a Quantum-Like Cognitive Architecture for Decision-Makingcited by 17
  35. 2020An Analysis of Dataset Overlap on Winograd-Style Taskscited by 16
  36. 2021Explanatory models in neuroscience: Part 2 -- constraint-based intelligibilitycited by 7
  37. 2022Norm Participation Grounds Languagecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 52; 2024: 59; 2025: 28; 2026: 14.

171 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  4. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  5. 2023Augmented Language Models: a Surveycited by 398
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  7. 2022Large Language Models Are Reasoning Teacherscited by 318
  8. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  9. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  12. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
38 more
  1. 2023Looped Transformers as Programmable Computerscited by 224
  2. 2023Playing repeated games with Large Language Modelscited by 213
  3. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  4. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  5. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  6. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  7. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  8. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  9. 2022Faithful Reasoning Using Large Language Modelscited by 130
  10. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 128
  11. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  12. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
  13. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  14. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  15. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  16. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  17. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  18. 2022Meaning without reference in large language modelscited by 76
  19. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  20. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  21. 2023Turning large language models into cognitive modelscited by 66
  22. 2024Attention Heads of Large Language Models: A Surveycited by 65
  23. 2023Natural Language Reasoning, A Surveycited by 61
  24. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  25. 2024CogBench: a large language model walks into a psychology labcited by 60
  26. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  27. 2023Meta-in-context learning in large language modelscited by 59
  28. 2024Can Language Models Learn to Skip Steps?cited by 59
  29. 2024Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluationcited by 56
  30. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  31. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  32. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 51
  33. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  34. 2023The language of prompting: What linguistic properties make a prompt successful?cited by 43
  35. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  36. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  37. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  38. 2023Instruction-tuning Aligns LLMs to the Human Braincited by 39

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Models
  2. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
  3. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  4. 2025Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning
  5. 2023Studying and improving reasoning in humans and machines
  6. 2023Concise and Organized Perception Facilitates Reasoning in Large Language Models
  7. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus
  8. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond