mlsanityData to 7 Oct 2026

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

arXiv 2304.03439 ↗First posted 7 Apr 2023cs.CL cs.AIcited by 202

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.

Key facts. Cited by 202 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 18 of its 25 references that are in the map. First posted on arXiv 7 Apr 2023.

Builds on

18 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2019Augmenting Neural Networks with First-order Logiccited by 51
  12. 2019Can neural networks understand monotonicity reasoning?cited by 45
5 more
  1. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  2. 2019HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoningcited by 35
  3. 2020TaxiNLI: Taking a Ride up the NLU Hillcited by 27
  4. 2020ConjNLI: Natural Language Inference Over Conjunctive Sentencescited by 27
  5. 2020Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Textscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 79; 2024: 78; 2025: 32; 2026: 13.

202 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  4. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  5. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2023How is ChatGPT's behavior changing over time?cited by 265
  8. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  10. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  11. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  12. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
38 more
  1. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  2. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  3. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  4. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  5. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  6. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  7. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 86
  8. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  9. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  10. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  11. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  12. 2024GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussioncited by 64
  13. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  14. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  15. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  16. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 53
  17. 2023Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformationcited by 51
  18. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  19. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  20. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  21. 2023Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framingcited by 49
  22. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  23. 2024Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Modelscited by 44
  24. 2025Rethinking Reflection in Pre-Trainingcited by 43
  25. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  26. 2023BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biologycited by 42
  27. 2023Exploring the Potential of Large Language Models in Computational Argumentationcited by 42
  28. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  29. 2023GPT-4 Can't Reasoncited by 39
  30. 2024Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Makingcited by 38
  31. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  32. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  33. 2023Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinationscited by 34
  34. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  35. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  36. 2024Large Language Models for Education: A Surveycited by 32
  37. 2023LogiCoT: Logical Chain-of-Thought Instruction-Tuningcited by 31
  38. 2023Exploring the Boundaries of GPT-4 in Radiologycited by 31

50 most cited of 202 All 202 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts
  2. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
  3. 2023Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing
  4. 2023Consistency Analysis of ChatGPT
  5. 2024LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
  6. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models
  7. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought
  8. 2023Using Language Models For Knowledge Acquisition in Natural Language Reasoning Problems