mlsanityData to 7 Oct 2026

GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts

arXiv 2305.12477 ↗First posted 21 May 2023cs.CL cs.AIcited by 34

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have exhibited remarkable performance on various Natural Language Processing (NLP) tasks. However, there is a current hot debate regarding their reasoning capacity. In this paper, we examine the performance of GPT-3.5, GPT-4, and BARD models, by performing a thorough technical evaluation on different reasoning tasks across eleven distinct datasets. Our paper provides empirical evidence showcasing the superior performance of ChatGPT-4 in comparison to both ChatGPT-3.5 and BARD in zero-shot setting throughout almost all evaluated tasks. While the superiority of GPT-4 compared to GPT-3.5 might be explained by its larger size and NLP efficiency, this was not evident for BARD. We also demonstrate that the three models show limited proficiency in Inductive, Mathematical, and Multi-hop Reasoning Tasks. To bolster our findings, we present a detailed and comprehensive analysis of the results from these three models. Furthermore, we propose a set of engineered prompts that enhances the zero-shot setting performance of all three models.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 17 of its 27 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

17 of 27 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  11. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  12. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
5 more
  1. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  2. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  3. 2019Abductive Commonsense Reasoningcited by 300
  4. 2021Explaining Answers with Entailment Treescited by 148
  5. 2022e-CARE: a New Dataset for Exploring Explainable Causal Reasoningcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 18; 2025: 8; 2026: 1.

34 papers.

Sort:

By area:

  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  2. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 73
  3. 2023LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMscited by 40
  4. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiencycited by 31
  5. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  6. 2024Large Language Models in Wireless Application Design: In-Context Learning-enhanced Automatic Network Intrusion Detectioncited by 16
  7. 2024Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awarenesscited by 14
  8. 2023Evaluation of large language models for discovery of gene set functioncited by 13
  9. 2024Can Textual Semantics Mitigate Sounding Object Segmentation Preference?cited by 13
  10. 2024LLMs in Biomedicine: A study on clinical Named Entity Recognitioncited by 12
  11. 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Studycited by 12
  12. 2023ChatGPT in the Age of Generative AI and Large Language Models: A Concise Surveycited by 10
22 more
  1. 2024Using Large Language Models to Assess Tutors' Performance in Reacting to Students Making Math Errorscited by 9
  2. 2025Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategiescited by 8
  3. 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AIcited by 7
  4. 2024TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidancecited by 5
  5. 2025Are Large Language Models Good at Detecting Propaganda?cited by 5
  6. 2024Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Examcited by 4
  7. 2023DocChecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsistenciescited by 3
  8. 2025Investigating Tax Evasion Emergence Using Dual Large Language Model and Deep Reinforcement Learning Powered Agent-based Simulationcited by 3
  9. 2023Benchmarking and Analyzing In-context Learning, Fine-tuning and Supervised Learning for Biomedical Knowledge Curation: a focused study on chemical entities of biological interestcited by 2
  10. 2024A Preliminary Study on Using Large Language Models in Software Pentestingcited by 2
  11. 2024Construction of Hyper-Relational Knowledge Graphs Using Pre-Trained Large Language Modelscited by 2
  12. 2024Can GPT-4 learn to analyse moves in research article abstracts?cited by 2
  13. 2025Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMscited by 2
  14. 2025Assessing the Capability of LLMs in Solving POSCOMP Questionscited by 2
  15. 2025Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehiclescited by 2
  16. 2024Evaluation of Language Models in the Medical Context Under Resource-Constrained Settingscited by 1
  17. 2025WIP: Assessing the Effectiveness of ChatGPT in Preparatory Testing Activitiescited by 1
  18. 2026Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Modelscited by 1
  19. 2023Large Language Models for Semantic Monitoring of Corporate Disclosures: A Case Study on Korea's Top 50 KOSPI Companiescited by 0
  20. 2024Assisting humans in complex comparisons: automated information comparison at scalecited by 0
  21. 2024ExIQA: Explainable Image Quality Assessment Using Distortion Attributescited by 0
  22. 2025Measuring Algorithmic Partisanship via Zero-Shot Classification and Its Implications on Political Discoursecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
  2. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
  3. 2022Prompting GPT-3 To Be Reliable
  4. 2025Reasoning Capabilities and Invariability of Large Language Models
  5. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models
  6. 2025GPT-5 Model Corrected GPT-4V's Chart Reading Errors, Not Prompting
  7. 2023An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
  8. 2023Pushing the Limits of ChatGPT on NLP Tasks