# RuozhiBench: Evaluating LLMs with Logical Fallacies and Misleading Premises

arXiv: https://arxiv.org/abs/2502.13125 · HTML: https://mlsanity.com/paper/2502.13125/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2025.

## Built on by (most cited first, 6 of 6)

- [Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models](https://mlsanity.com/paper/2505.23715/) (2025)
- [Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them](https://mlsanity.com/paper/2503.16401/) (2025)
- [Socrates or Smartypants: Testing Logic Reasoning Capabilities of Large Language Models with Logic Programming-based Test Oracles](https://mlsanity.com/paper/2504.12312/) (2025)
- [MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data](https://mlsanity.com/paper/2510.26345/) (2025)
- [DeflectBench: A Benchmark for Evaluating Rhetorical Fallacy Generation in LLMs](https://mlsanity.com/paper/2608.26119/) (2026)
- [Beyond Routine Compliance: Cunning Data Cultivates Safety Vigilance in Large Language Models](https://mlsanity.com/paper/2609.18515/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have shown that they can answer questions requiring complex reasoning. However, their ability to identify and respond to text containing logical fallacies or deliberately misleading premises remains less studied. To address this gap, we introduce RuozhiBench, a bilingual dataset comprising 677 carefully curated questions that contain various forms of deceptive reasoning, meticulously crafted through extensive human effort and expert review. In a comprehensive evaluation of 17 LLMs from 5 Series over RuozhiBench using both open-ended and two-choice formats, we conduct extensive analyses on evaluation protocols and result patterns. Despite their high scores on conventional benchmarks, these models showed limited ability to detect and reason correctly about logical fallacies, with even the best-performing model, Claude-3-haiku, achieving only 62% accuracy compared to the human of more than 90%.