mlsanityData to 7 Oct 2026

Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning

arXiv 2510.24435 ↗First posted 28 Oct 2025cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Evaluating reasoning ability in Large Language Models (LLMs) is important for advancing artificial intelligence, as it transcends mere linguistic task performance. It involves understanding whether these models truly understand information, perform inferences, and are able to draw conclusions in a logical and valid way. This study compare logical and abstract reasoning skills of several LLMs - including GPT, Claude, DeepSeek, Gemini, Grok, Llama, Mistral, Perplexity, and Sabi'a - using a set of eight custom-designed reasoning questions. The LLM results are benchmarked against human performance on the same tasks, revealing significant differences and indicating areas where LLMs struggle with deduction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 6 of its 12 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

6 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Lost in the Logic: An Evaluation of Large Language Models' Reasoning Capabilities on LSAT Logic Games23 Sep

Other references · 5

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  4. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  5. 2021Abstract Reasoning via Logic-guided Generationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus
  2. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
  3. 2022Towards Reasoning in Large Language Models: A Survey
  4. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Models
  6. 2025Logical Reasoning in Large Language Models: A Survey
  7. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  8. 2023Large Language Models Are Not Strong Abstract Reasoners