mlsanityData to 7 Oct 2026

LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

arXiv 2007.08124 ↗First posted 16 Jul 2020cs.CLcited by 68

Not on the map. Why?

Cited by 61 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Machine reading is a fundamental task for testing the capability of natural language understanding, which is closely related to human cognition in many aspects. With the rising of deep learning techniques, algorithmic models rival human performances on simple QA, and thus increasingly challenging machine reading datasets have been proposed. Though various challenges such as evidence integration and commonsense knowledge have been integrated, one of the fundamental capabilities in human reading, namely logical reasoning, is not fully investigated. We build a comprehensive dataset, named LogiQA, which is sourced from expert-written questions for testing human Logical reasoning. It consists of 8,678 QA instances, covering multiple types of deductive reasoning. Results show that state-of-the-art neural models perform by far worse than human ceiling. Our dataset can also serve as a benchmark for reinvestigating logical AI under the deep learning NLP setting. The dataset is freely available at https://github.com/lgw863/LogiQA-dataset

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 32; 2026: 24.

68 papers.

Sort:

By area:

  1. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  2. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  3. 2024Understanding Chain-of-Thought in LLMs through Information Theorycited by 36
  4. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacycited by 34
  5. 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Studycited by 30
  6. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 29
  7. 2025JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Modelscited by 21
  8. 2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 17
  9. 2025ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problemscited by 17
  10. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 16
  11. 2024Optimizing Language Model's Reasoning Abilities with Weak Supervisioncited by 15
  12. 2025CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesiscited by 15
38 more
  1. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
  2. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Reviewcited by 14
  3. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  4. 2025Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shotcited by 12
  5. 2024Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Modelscited by 11
  6. 2026Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimizationcited by 9
  7. 2025Reasoning with Graphs: Structuring Implicit Knowledge to Enhance LLMs Reasoningcited by 8
  8. 2025Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?cited by 7
  9. 2025PrefDisco: Benchmarking Proactive Personalized Reasoningcited by 7
  10. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 7
  11. 2025Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Modelscited by 6
  12. 2025Reason from Future: Reverse Thought Chain Enhances LLM Reasoningcited by 5
  13. 2025Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?cited by 5
  14. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RLcited by 5
  15. 2025On Code-Induced Reasoning in LLMscited by 5
  16. 2025Correct Reasoning Paths Visit Shared Decision Pivotscited by 5
  17. 2025DART: Difficulty-Adaptive Reasoning Truncation for Efficient Large Language Modelscited by 5
  18. 2024Improving Language Model Reasoning with Self-motivated Learningcited by 4
  19. 2025MastermindEval: A Simple But Scalable Reasoning Benchmarkcited by 4
  20. 2024Thought-Path Contrastive Learning via Premise-Oriented Data Augmentation for Logical Reading Comprehensioncited by 3
  21. 2024Can OpenAI o1 outperform humans in higher-order cognitive thinking?cited by 3
  22. 2025Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoningcited by 3
  23. 2025CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Densitycited by 3
  24. 2026How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domainscited by 3
  25. 2026Detecting Unfaithful Chain-of-Thought via Circuit-Guided Internal-External Discrepancycited by 3
  26. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scalingcited by 3
  27. 2025Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuningcited by 2
  28. 2026Rethinking Data Mixing from the Perspective of Large Language Modelscited by 2
  29. 2026Prefix Sliding for efficient test-time scalingcited by 2
  30. 2024PathReasoner: Modeling Reasoning Path with Equivalent Extension for Logical Question Answeringcited by 1
  31. 2025Computational Reasoning of Large Language Modelscited by 1
  32. 2025Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Modelscited by 1
  33. 2025Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoningcited by 1
  34. 2025Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspectivecited by 1
  35. 2025Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?cited by 1
  36. 2025EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Modelscited by 1
  37. 2026LogicGraph : Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verificationcited by 1
  38. 2026How Emotion Shapes the Behavior of LLMs and Agents: A Mechanistic Studycited by 1

50 most cited of 68 All 68 (JSON)