mlsanityData to 14 Oct 2026

Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus

arXiv 2411.12498 ↗First posted 19 Nov 2024cs.LG cs.AI cs.LOcited by 29

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) are capable of solving a wide range of tasks, yet they have struggled with reasoning. To address this, we propose Additional Logic Training (ALT), which aims to enhance LLMs' reasoning capabilities by program-generated logical reasoning samples. We first establish principles for designing high-quality samples by integrating symbolic logic theory and previous empirical insights. Then, based on these principles, we construct a synthetic corpus named Formal Logic Deduction Diverse (FLD× 2), comprising numerous samples of multi-step deduction with unknown facts, diverse reasoning rules, diverse linguistic expressions, and challenging distractors. Finally, we empirically show that ALT on FLD×2 substantially enhances the reasoning capabilities of state-of-the-art LLMs, including LLaMA-3.1-70B. Improvements include gains of up to 30 points on logical reasoning benchmarks, up to 10 points on math and coding benchmarks, and 5 points on the benchmark suite BBH.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 78 of its 123 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

78 of 123 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance?9 Oct
  2. 2024LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning19 Sep
  3. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset8 Aug
  4. 2024First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning23 Jun
  5. 2024Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models18 Jun
  6. 2024A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences17 Jun
  7. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners16 Jun
  8. 2024Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars16 Jun
  9. 2024Large Language Models are Limited in Out-of-Context Knowledge Reasoning11 Jun
  10. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs18 Feb
  11. 2023UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations14 Nov
  12. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning14 Nov
12 more
  1. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning24 Oct
  2. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks29 Sep
  3. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?28 Sep
  4. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic11 Aug
  5. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  6. 2023Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases21 Jun
  7. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
  8. 2022Can Pretrained Language Models (Yet) Reason Deductively?12 Oct
  9. 2022Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation28 Jul
  10. 2022On the Paradox of Learning to Reason from Data23 May
  11. 2022AbductionRules: Training Transformers to Explain Unexpected Inputs23 Mar
  12. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Language19 Mar

Other references · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  10. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  11. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  12. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  13. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  14. 2022Large Language Models Are Reasoning Teacherscited by 329
  15. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  16. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  17. 2019Abductive Commonsense Reasoningcited by 301
  18. 2022Teaching Small Language Models to Reasoncited by 284
  19. 2020Transformers as Soft Reasoners over Languagecited by 270
  20. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  21. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  22. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  23. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  24. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  25. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  26. 2021Explaining Answers with Entailment Treescited by 152
  27. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  28. 2023SCOTT: Self-Consistent Chain-of-Thought Distillationcited by 105
  29. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  30. 2017The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrantscited by 56
  31. 2024Don't Trust: Verify -- Grounding LLM Quantitative Reasoning with Autoformalizationcited by 55
  32. 2020PRover: Proof Generation for Interpretable Reasoning over Rulescited by 53
  33. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 51
  34. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  35. 2019HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoningcited by 35
  36. 2020Critical Thinking for Language Modelscited by 31
  37. 2017Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussionscited by 30
  38. 2021Flexible Generation of Natural Language Deductionscited by 28

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 11.

29 papers.

Sort:

By area:

  1. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  2. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  3. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  4. 2025Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Modelscited by 10
  5. 2025Efficient PRM Training Data Synthesis via Formal Verificationcited by 8
  6. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  7. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  8. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Frameworkcited by 5
  9. 2025LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertaintycited by 5
  10. 2026NL2LOGIC: AST-Guided Translation of Natural Language into First-Order Logic with Large Language Modelscited by 5
  11. 2025Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretationscited by 4
  12. 2025LM-mixup: Text Data Augmentation via Language Model based Mixupcited by 3
17 more
  1. 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
  2. 2025Matching Markets Meet LLMs: Algorithmic Reasoning with Ranked Preferencescited by 2
  3. 2025LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Modelscited by 2
  4. 2025RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learningcited by 2
  5. 2026Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibilitycited by 2
  6. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  7. 2026Position: Logical Soundness is not a Reliable Criterion for Neurosymbolic Fact-Checking with LLMscited by 1
  8. 2026Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesiscited by 1
  9. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  10. 2025Quantifying Logical Consistency in Transformers via Query-Key Alignmentcited by 0
  11. 2025Probing AI Safety with Source Codecited by 0
  12. 2025On the Measure of a Model: From Intelligence to Generalitycited by 0
  13. 2026FloCA: Towards Faithful and Logically Consistent Flowchart Reasoningcited by 0
  14. 2026Localizing Input Uncertainty Quantification for Large Language Models via Shapley Valuescited by 0
  15. 2026Multi-Adapter Representation Interventions via Energy Calibrationcited by 0
  16. 2026From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligencecited by 0
  17. 2026Natural Language to First-Order Logic LLM-based Autoformalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach
  2. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  3. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
  4. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  5. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
  6. 2024LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations
  7. 2025SLR: Automated Synthesis for Scalable Logical Reasoning
  8. 2025Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning