mlsanityData to 14 Oct 2026

Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate

arXiv 2305.13160 ↗First posted 22 May 2023cs.CLcited by 94

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) such as ChatGPT and GPT-4 have shown impressive performance in complex reasoning tasks. However, it is difficult to know whether the models are reasoning based on deep understandings of truth and logic, or leveraging their memorized patterns in a relatively superficial way. In this work, we explore testing LLMs' reasoning by engaging with them in a debate-like conversation, where given a question, the LLM and the user need to discuss to make the correct decision starting from opposing arguments. Upon mitigating the Clever Hans effect, our task requires the LLM to not only achieve the correct answer on its own, but also be able to hold and defend its belief instead of blindly believing or getting misled by the user's (invalid) arguments and critiques, thus testing in greater depth whether the LLM grasps the essence of the reasoning required to solve the problem. Across a range of complex reasoning benchmarks spanning math, commonsense, logic and BIG-Bench tasks, we find that despite their impressive performance as reported in existing work on generating correct step-by-step solutions in the beginning, LLMs like ChatGPT cannot maintain their beliefs in truth for a significant portion of examples when challenged by oftentimes absurdly invalid arguments. Our work points to danger zones of model alignment, and also suggests more careful treatments and interpretations of the recent findings that LLMs can improve their responses based on feedback.

Key facts. Cited by 94 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 41 of its 64 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

41 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  2. 2022Can Pretrained Language Models (Yet) Reason Deductively?12 Oct
  3. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  4. 2022RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning25 May

Other references · 37

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  11. 2023Teaching Large Language Models to Self-Debugcited by 988
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
25 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  2. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  4. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  5. 2018AI safety via debatecited by 400
  6. 2023Language Models can Solve Computer Taskscited by 372
  7. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  8. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  9. 2022Data Contamination: From Memorization to Exploitationcited by 236
  10. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  11. 2018Stress Test Evaluation for Natural Language Inferencecited by 213
  12. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  13. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 207
  14. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  15. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  16. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  17. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  18. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  19. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  20. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  21. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  22. 2022Rainier: Reinforced Knowledge Introspector for Commonsense Question Answeringcited by 43
  23. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 37
  24. 2023Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMscited by 20
  25. 2023"Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Processcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 30; 2025: 41; 2026: 16.

94 papers.

Sort:

By area:

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  3. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  4. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  5. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  6. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  7. 2024Do LLMs write like humans? Variation in grammatical and rhetorical stylescited by 62
  8. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  9. 2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 47
  10. 2023Debate Helps Supervise Unreliable Expertscited by 43
  11. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  12. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 34
38 more
  1. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  2. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 24
  3. 2024REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Modelscited by 23
  4. 2024A Survey on the Honesty of Large Language Modelscited by 23
  5. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  6. 2025Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agentscited by 22
  7. 2025Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamicscited by 19
  8. 2025Enhancing Multi-Agent Debate System Performance via Confidence Expressioncited by 19
  9. 2024Assistive Large Language Model Agents for Socially-Aware Negotiation Dialoguescited by 18
  10. 2025Challenging the Evaluator: LLM Sycophancy Under User Rebuttalcited by 18
  11. 2024EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generationcited by 17
  12. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  13. 2024A Survey on Human-Centric LLMscited by 13
  14. 2023Evaluating Large Language Models for Health-related Queries with Presuppositionscited by 12
  15. 2025Feedback Friction: LLMs Struggle to Fully Incorporate External Feedbackcited by 11
  16. 2025Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectoriescited by 11
  17. 2025Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?cited by 10
  18. 2024A Linguistic Comparison between Human and ChatGPT-Generated Conversationscited by 9
  19. 2024Unlocking the Power of LLM Uncertainty for Active In-Context Example Selectioncited by 9
  20. 2025LLMs for Customized Marketing Content Generation and Evaluation at Scalecited by 9
  21. 2025Tailored Truths: Optimizing LLM Persuasion with Personalization and Fabricated Statisticscited by 8
  22. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 8
  23. 2025Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Modelscited by 8
  24. 2025CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Taskscited by 8
  25. 2024Models That Prove Their Own Correctnesscited by 7
  26. 2025Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notescited by 7
  27. 2025Don't Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMscited by 7
  28. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  29. 2023Playing Large Games with Oracles and AI Debatecited by 6
  30. 2024Antagonistic AIcited by 6
  31. 2025Benchmarking Gaslighting Negation Attacks Against Reasoning Modelscited by 6
  32. 2023XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMscited by 5
  33. 2023Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Modelscited by 5
  34. 2024"My Grade is Wrong!": A Contestable AI Framework for Interactive Feedback in Evaluating Student Essayscited by 5
  35. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humanscited by 5
  36. 2024Fostering Natural Conversation in Large Language Models with NICO: a Natural Interactive COnversation datasetcited by 4
  37. 2025RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaborationcited by 4
  38. 2025AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefscited by 4

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying
  2. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
  4. 2025Can LLMs Judge Debates? Evaluating Non-Linear Reasoning via Argumentation Theory Semantics
  5. 2024Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
  6. 2024CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge
  7. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Survey
  8. 2025What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation