# Measuring reasoning capabilities of ChatGPT

arXiv: https://arxiv.org/abs/2310.05993 · HTML: https://mlsanity.com/paper/2310.05993/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 0 of its 10 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Built on by (most cited first, 2 of 2)

- [Solving Zebra Puzzles Using Constraint-Guided Multi-Agent Systems](https://mlsanity.com/paper/2407.03956/) (2024)
- [A Statistical Analysis of LLMs' Self-Evaluation Using Proverbs](https://mlsanity.com/paper/2410.16640/) (2024)

## Abstract (arXiv)

I shall quantify the logical faults generated by ChatGPT when applied to reasoning tasks. For experiments, I use the 144 puzzles from the library urlhttps://users.utcluj.ro/~agroza/puzzles/maloga~citegroza:fol. The library contains puzzles of various types, including arithmetic puzzles, logical equations, Sudoku-like puzzles, zebra-like puzzles, truth-telling puzzles, grid puzzles, strange numbers, or self-reference puzzles. The correct solutions for these puzzles were checked using the theorem prover Prover9~citemccune2005release and the finite models finder Mace4~citemccune2003mace4 based on human-modelling in Equational First Order Logic. A first output of this study is the benchmark of 100 logical puzzles. For this dataset ChatGPT provided both correct answer and justification for 7% only. %, while BARD for 5%. Since the dataset seems challenging, the researchers are invited to test the dataset on more advanced or tuned models than ChatGPT3.5 with more crafted prompts. A second output is the classification of reasoning faults conveyed by ChatGPT. This classification forms a basis for a taxonomy of reasoning faults generated by large language models. I have identified 67 such logical faults, among which: inconsistencies, implication does not hold, unsupported claim, lack of commonsense, wrong justification. The 100 solutions generated by ChatGPT contain 698 logical faults. That is on average, 7 fallacies for each reasoning task. A third ouput is the annotated answers of the ChatGPT with the corresponding logical faults. Each wrong statement within the ChatGPT answer was manually annotated, aiming to quantify the amount of faulty text generated by the language model. On average, 26.03% from the generated text was a logical fault.