mlsanityData to 7 Oct 2026

FOLIO: Natural Language Reasoning with First-Order Logic

arXiv 2209.00840 ↗First posted 2 Sep 2022cs.CLcited by 233

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. The logical correctness of the premises and conclusions is ensured by their FOL annotations, which are automatically verified by an FOL inference engine. In addition to the main NL reasoning task, NL-FOL pairs in FOLIO constitute a new NL-FOL translation dataset. Our experiments on FOLIO systematically evaluate the FOL reasoning ability of supervised fine-tuning on medium-sized language models. For both NL reasoning and NL-FOL translation, we benchmark multiple state-of-the-art language models. Our results show that a subset of FOLIO presents a challenge for one of the most capable {Large Language Model (LLM)} publicly available, GPT-4.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 45 of its 106 references that are in the map. First posted on arXiv 2 Sep 2022.

Builds on

45 of 106 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Language models show human-like content effects on reasoning tasks14 Jul
  2. 2022On the Paradox of Learning to Reason from Data23 May
  3. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May

Other references · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
30 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  8. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  9. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  10. 2020Transformers as Soft Reasoners over Languagecited by 265
  11. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  12. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  13. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  14. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  15. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  16. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  17. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  18. 2022Language Model Cascadescited by 104
  19. 2020Logical Natural Language Generation from Open-Domain Tablescited by 90
  20. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 65
  21. 2020Teaching Temporal Logics to Neural Networkscited by 51
  22. 2019Can neural networks understand monotonicity reasoning?cited by 45
  23. 2019HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoningcited by 35
  24. 2020Critical Thinking for Language Modelscited by 31
  25. 2020RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axiomscited by 28
  26. 2021Flexible Generation of Natural Language Deductionscited by 28
  27. 2021RuleBert: Teaching Soft Rules to Pre-trained Language Modelscited by 25
  28. 2021Database Reasoning Over Textcited by 20
  29. 2020Exploring Neural Models for Parsing Natural Language into First-Order Logiccited by 16
  30. 2022Uncovering More Shallow Heuristics: Probing the Natural Language Inference Capacities of Transformer-Based Pre-Trained Language Models Using Syllogistic Patternscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 30; 2024: 55; 2025: 89; 2026: 56.

233 papers.

Sort:

By area:

  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  3. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  4. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  5. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  6. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  7. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  8. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  9. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  11. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  12. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
38 more
  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  2. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  3. 2023Cumulative Reasoning with Large Language Modelscited by 94
  4. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  5. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  6. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  7. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  8. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  9. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  10. 2023Natural Language Reasoning, A Surveycited by 61
  11. 2023Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translationcited by 59
  12. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 53
  13. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  14. 2025A Survey of Large Language Model Agents for Question Answeringcited by 46
  15. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  16. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Modelscited by 38
  17. 2025CRANE: Reasoning with constrained LLM generationcited by 38
  18. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  19. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacycited by 34
  20. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  21. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Surveycited by 32
  22. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  23. 2023LogiCoT: Logical Chain-of-Thought Instruction-Tuningcited by 31
  24. 2025Steering LLM Thinking with Budget Guidancecited by 30
  25. 2024Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMscited by 29
  26. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 29
  27. 2024LeanReasoner: Boosting Complex Logical Reasoning with Leancited by 28
  28. 2023SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluationcited by 27
  29. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoningcited by 27
  30. 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoningcited by 27
  31. 2025Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluationcited by 27
  32. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 26
  33. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  34. 2025TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialoguescited by 25
  35. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 24
  36. 2024Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Modelscited by 23
  37. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  38. 2025Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMscited by 22

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification
  2. 2024Autoformalizing Natural Language to First-Order Logic: A Case Study in Logical Fallacy Detection
  3. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  4. 2026Improving Symbolic Translation of Language Models for Logical Reasoning
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2025Efficient Inference for Large Reasoning Models: A Survey
  7. 2025Assessing the Sensitivity and Alignment of FOL Closeness Metrics
  8. 2022Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models