mlsanityData to 7 Oct 2026

Improved Logical Reasoning of Language Models via Differentiable Symbolic Programming

arXiv 2305.03742 ↗First posted 5 May 2023cs.AI cs.CL cs.LGcited by 18

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Pre-trained large language models (LMs) struggle to perform logical reasoning reliably despite advances in scale and compositionality. In this work, we tackle this challenge through the lens of symbolic programming. We propose DSR-LM, a Differentiable Symbolic Reasoning framework where pre-trained LMs govern the perception of factual knowledge, and a symbolic module performs deductive reasoning. In contrast to works that rely on hand-crafted logic rules, our differentiable symbolic reasoning framework efficiently learns weighted rules and applies semantic loss to further improve LMs. DSR-LM is scalable, interpretable, and allows easy integration of prior knowledge, thereby supporting extensive symbolic programming to robustly derive a logical conclusion. The results of our experiments suggest that DSR-LM improves the logical reasoning abilities of pre-trained language models, resulting in a significant increase in accuracy of over 20% on deductive reasoning benchmarks. Furthermore, DSR-LM outperforms a variety of competitive baselines when faced with systematic changes in sequence length.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 44 of its 60 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

44 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Evaluating Step-by-Step Reasoning through Symbolic Verification16 Dec
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  3. 2022On the Paradox of Learning to Reason from Data23 May
  4. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May

Other references · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2017Graph Attention Networkscited by 8,854
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
28 more
  1. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 544
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 389
  6. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  7. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  9. 2017A simple neural network module for relational reasoningcited by 345
  10. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 271
  11. 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 267
  12. 2020Transformers as Soft Reasoners over Languagecited by 265
  13. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  14. 2017Learning Explanatory Rules from Noisy Datacited by 210
  15. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  16. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  17. 2017End-to-End Differentiable Provingcited by 159
  18. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  19. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  20. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  21. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  22. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  23. 2020Learning Reasoning Strategies in End-to-End Differentiable Provingcited by 60
  24. 2020Efficient Probabilistic Logic Reasoning with Graph Neural Networkscited by 50
  25. 2020Critical Thinking for Language Modelscited by 31
  26. 2020Towards Interpretable Natural Language Understanding with Explanations as Latent Variablescited by 30
  27. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30
  28. 2021RuleBert: Teaching Soft Rules to Pre-trained Language Modelscited by 25

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 6; 2025: 5.

18 papers.

Sort:

By area:

  1. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  2. 2023Large Language Models as Analogical Reasonerscited by 107
  3. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  4. 2023Large Language Models can Learn Rulescited by 66
  5. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  6. 2024Logically Consistent Language Models via Neuro-Symbolic Integrationcited by 30
  7. 2024XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Modelscited by 23
  8. 2025Constrained Discrete Diffusioncited by 16
  9. 2024Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understandingcited by 12
  10. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  11. 2023Teaching Probabilistic Logical Reasoning to Transformerscited by 7
  12. 2023Enhancing Logical Reasoning in Large Language Models to Facilitate Legal Applicationscited by 6
6 more
  1. 2025A Comparative Study of Neurosymbolic AI Approaches to Interpretable Logical Reasoningcited by 4
  2. 2025ESCA: Contextualizing Embodied Agents via Scene-Graph Generationcited by 4
  3. 2024Imbalances in Neurosymbolic Learning: Characterization and Mitigating Strategiescited by 2
  4. 2024DANA: Domain-Aware Neurosymbolic Agents for Consistency and Accuracycited by 2
  5. 2025Neuro-Symbolic Frameworks: Conceptual Characterization and Empirical Comparative Analysiscited by 2
  6. 2025Order Doesn't Matter, But Reasoning Does: Training LLMs with Order-Centric Augmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
  2. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  3. 2024LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations
  4. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach
  5. 2024Reliable Reasoning Beyond Natural Language
  6. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus
  7. 2026Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification
  8. 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs