mlsanityData to 7 Oct 2026

GLoRE: Evaluating Logical Reasoning of Large Language Models

arXiv 2310.09107 ↗First posted 13 Oct 2023cs.CL cs.AIcited by 13

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have shown significant general language understanding abilities. However, there has been a scarcity of attempts to assess the logical reasoning capacities of these LLMs, an essential facet of natural language understanding. To encourage further investigation in this area, we introduce GLoRE, a General Logical Reasoning Evaluation platform that not only consolidates diverse datasets but also standardizes them into a unified format suitable for evaluating large language models across zero-shot and few-shot scenarios. Our experimental results show that compared to the performance of humans and supervised fine-tuning models, the logical reasoning capabilities of large reasoning models, such as OpenAI's o1 mini, DeepSeek R1 and QwQ-32B, have seen remarkable improvements, with QwQ-32B achieving the highest benchmark performance to date. GLoRE is designed as a living project that continuously integrates new datasets and models, facilitating robust and comparative assessments of model performance in both commercial and Huggingface communities.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 65 of its 83 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

65 of 83 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  2. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance26 May
  3. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples24 May
  4. 2023PaLM 2 Technical Report17 May
  5. 2023ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness21 Apr
  6. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb
  8. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  9. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Language19 Mar
  10. 2022Logical Reasoning for Task Oriented Dialogue Systems8 Feb

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2023BloombergGPT: A Large Language Model for Financecited by 923
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  14. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  15. 2022A Survey on In-context Learningcited by 717
  16. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  17. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  18. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  19. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  20. 2020Transformers as Soft Reasoners over Languagecited by 265
  21. 2023Mathematical Capabilities of ChatGPTcited by 263
  22. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  23. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  24. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  25. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  26. 2022Can large language models reason about medical questions?cited by 194
  27. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  28. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  29. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114
  30. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  31. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 71
  32. 2023Could a Large Language Model be Conscious?cited by 70
  33. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  34. 2020PRover: Proof Generation for Interpretable Reasoning over Rulescited by 53
  35. 2019Augmenting Neural Networks with First-order Logiccited by 51
  36. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  37. 2019HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoningcited by 35
  38. 2023LogiCoT: Logical Chain-of-Thought Instruction-Tuningcited by 31

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 3; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Modelscited by 39
  2. 2024Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?cited by 25
  3. 2024A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferencescited by 13
  4. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  5. 2024Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answerscited by 9
  6. 2024SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoningcited by 9
  7. 2024AI Benchmarks and Datasets for LLM Evaluationcited by 7
  8. 2025Evaluating the Logical Reasoning Abilities of Large Reasoning Modelscited by 7
  9. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Frameworkcited by 4
  10. 2025Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoningcited by 2
  11. 2026Scaling the Scaling Logic: Agentic Meta-Synthesis of Logic Reasoningcited by 1
  12. 2026SCRuB: Social Concept Reasoning under Rubric-Based Evaluationcited by 1
1 more
  1. 2026Think Deep, Speak Once: Relit, A Recursive Latent Implicit Transformer Frameworkcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2025Logical Reasoning in Large Language Models: A Survey
  4. 2023Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning
  5. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  6. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  7. 2025Towards Reasoning Ability of Small Language Models
  8. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models