# LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

arXiv: https://arxiv.org/abs/2007.08124 · HTML: https://mlsanity.com/paper/2007.08124/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2020.

## Built on by (most cited first, 30 of 69)

- [Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking](https://mlsanity.com/paper/2502.13842/) (2025)
- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [Understanding Chain-of-Thought in LLMs through Information Theory](https://mlsanity.com/paper/2411.11984/) (2024)
- [DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy](https://mlsanity.com/paper/2310.18659/) (2023)
- [AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models](https://mlsanity.com/paper/2502.16906/) (2025)
- [Assessing Judging Bias in Large Reasoning Models: An Empirical Study](https://mlsanity.com/paper/2504.09946/) (2025)
- [JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models](https://mlsanity.com/paper/2501.14851/) (2025)
- [Cross-Modal Consistency in Multimodal Large Language Models](https://mlsanity.com/paper/2411.09273/) (2024)
- [ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems](https://mlsanity.com/paper/2507.04766/) (2025)
- [AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Models](https://mlsanity.com/paper/2505.17312/) (2025)
- [CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis](https://mlsanity.com/paper/2503.23145/) (2025)
- [Optimizing Language Model's Reasoning Abilities with Weak Supervision](https://mlsanity.com/paper/2405.04086/) (2024)
- [Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations](https://mlsanity.com/paper/2403.14112/) (2024)
- [Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review](https://mlsanity.com/paper/2410.03663/) (2024)
- [Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://mlsanity.com/paper/2404.04293/) (2024)
- [Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models](https://mlsanity.com/paper/2408.05093/) (2024)
- [Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot](https://mlsanity.com/paper/2506.14641/) (2025)
- [Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization](https://mlsanity.com/paper/2602.02188/) (2026)
- [Reasoning with Graphs: Structuring Implicit Knowledge to Enhance LLMs Reasoning](https://mlsanity.com/paper/2501.07845/) (2025)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?](https://mlsanity.com/paper/2507.04391/) (2025)
- [PrefDisco: Benchmarking Proactive Personalized Reasoning](https://mlsanity.com/paper/2510.00177/) (2025)
- [Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Models](https://mlsanity.com/paper/2505.17225/) (2025)
- [Reason from Future: Reverse Thought Chain Enhances LLM Reasoning](https://mlsanity.com/paper/2506.03673/) (2025)
- [Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?](https://mlsanity.com/paper/2506.04575/) (2025)
- [Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL](https://mlsanity.com/paper/2509.06024/) (2025)
- [On Code-Induced Reasoning in LLMs](https://mlsanity.com/paper/2509.21499/) (2025)
- [Correct Reasoning Paths Visit Shared Decision Pivots](https://mlsanity.com/paper/2509.21549/) (2025)
- [DART: Difficulty-Adaptive Reasoning Truncation for Efficient Large Language Models](https://mlsanity.com/paper/2511.01170/) (2025)
- [Improving Language Model Reasoning with Self-motivated Learning](https://mlsanity.com/paper/2404.07017/) (2024)

## Abstract (arXiv)

Machine reading is a fundamental task for testing the capability of natural language understanding, which is closely related to human cognition in many aspects. With the rising of deep learning techniques, algorithmic models rival human performances on simple QA, and thus increasingly challenging machine reading datasets have been proposed. Though various challenges such as evidence integration and commonsense knowledge have been integrated, one of the fundamental capabilities in human reading, namely logical reasoning, is not fully investigated. We build a comprehensive dataset, named LogiQA, which is sourced from expert-written questions for testing human Logical reasoning. It consists of 8,678 QA instances, covering multiple types of deductive reasoning. Results show that state-of-the-art neural models perform by far worse than human ceiling. Our dataset can also serve as a benchmark for reinvestigating logical AI under the deep learning NLP setting. The dataset is freely available at https://github.com/lgw863/LogiQA-dataset