mlsanityData to 14 Oct 2026

ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

arXiv 2002.04326 ↗First posted 11 Feb 2020cs.CL cs.AI cs.LGcited by 217

Not on the map. Why?

Cited by 121 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.

Key facts. Cited by 217 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 27; 2023: 46; 2024: 43; 2025: 45; 2026: 45.

217 papers.

Sort:

By area:

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2023Detecting Language Model Attacks with Perplexitycited by 431
  3. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  4. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  5. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
  6. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  7. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  8. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  9. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  10. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  11. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  12. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
38 more
  1. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 117
  2. 2022On the Paradox of Learning to Reason from Datacited by 93
  3. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  4. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 75
  5. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 69
  6. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 68
  7. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  8. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  9. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  10. 2023Natural Language Reasoning, A Surveycited by 61
  11. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 61
  12. 2023Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translationcited by 59
  13. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  14. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 53
  15. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 51
  16. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  17. 2023RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thoughtcited by 47
  18. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  19. 2021Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Textcited by 46
  20. 2022Reasoning Like Program Executorscited by 45
  21. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  22. 2023Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systemscited by 41
  23. 2024MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problemscited by 41
  24. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 37
  25. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  26. 2021Commonsense-Focused Dialogues for Response Generation: An Empirical Studycited by 30
  27. 2023MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarizationcited by 30
  28. 2024Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMscited by 29
  29. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  30. 2021DAGN: Discourse-Aware Graph Network for Logical Reasoningcited by 28
  31. 2021Flexible Generation of Natural Language Deductionscited by 28
  32. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28
  33. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  34. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  35. 2021English Machine Reading Comprehension Datasets: A Surveycited by 26
  36. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  37. 2024Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluationcited by 25
  38. 2024100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instancescited by 24

50 most cited of 217 All 217 (JSON)