mlsanityData to 7 Oct 2026

Can Large Language Models Act as Symbolic Reasoners?

arXiv 2410.21490 ↗First posted 28 Oct 2024cs.CL cs.AI cs.ETcited by 8

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The performance of Large language models (LLMs) across a broad range of domains has been impressive but have been critiqued as not being able to reason about their process and conclusions derived. This is to explain the conclusions draw, and also for determining a plan or strategy for their approach. This paper explores the current research in investigating symbolic reasoning and LLMs, and whether an LLM can inherently provide some form of reasoning or whether supporting components are necessary, and, if there is evidence for a reasoning capability, is this evident in a specific domain or is this a general capability? In addition, this paper aims to identify the current research gaps and future trends of LLM explainability, presenting a review of the literature, identifying current research into this topic and suggests areas for future work.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 32 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

19 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning2 Feb
  2. 2023Code Llama: Open Foundation Models for Code24 Aug
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct
  4. 2022On the Paradox of Learning to Reason from Data23 May
  5. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks11 May

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  8. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  9. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  10. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  11. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  12. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
2 more
  1. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  2. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  2. 2025HoarePrompt: Structural Reasoning About Program Correctness in Natural Languagecited by 7
  3. 2025AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Modelscited by 5
  4. 2025Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Modelscited by 2
  5. 2026Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshapingcited by 1
  6. 2025Transformers Can Learn Connectivity in Some Graphs but Not Otherscited by 0
  7. 2026State Design Matters: How Representations Shape Dynamic Reasoning in Large Language Modelscited by 0
  8. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  2. 2022Towards Reasoning in Large Language Models: A Survey
  3. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  4. 2024Do Large Language Models Understand Logic or Just Mimick Context?
  5. 2026Large Language Model Reasoning Failures
  6. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
  7. 2025Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives
  8. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey