mlsanityData to 14 Oct 2026

Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text

arXiv 2307.07696 ↗First posted 15 Jul 2023cs.CL cs.AI cs.SCcited by 59

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

While large language models (LLMs), such as GPT-3, appear to be robust and general, their reasoning ability is not at a level to compete with the best models trained for specific natural language reasoning problems. In this study, we observe that a large language model can serve as a highly effective few-shot semantic parser. It can convert natural language sentences into a logical form that serves as input for answer set programs, a logic-based declarative knowledge representation formalism. The combination results in a robust and general system that can handle multiple question-answering tasks without requiring retraining for each new task. It only needs a few examples to guide the LLM's adaptation to a specific task, along with reusable ASP knowledge modules that can be applied to multiple tasks. We demonstrate that this method achieves state-of-the-art performance on several NLP benchmarks, including bAbI, StepGame, CLUTRR, and gSCAN. Additionally, it successfully tackles robot planning tasks that an LLM alone fails to solve.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 43 of its 120 references that are in the map. First posted on arXiv 15 Jul 2023.

Builds on

43 of 120 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  7. 2018Universal Transformerscited by 647
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  10. 2017A simple neural network module for relational reasoningcited by 346
  11. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  12. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
31 more
  1. 2018Deep Learning: A Critical Appraisalcited by 217
  2. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  3. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  4. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  5. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 163
  6. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  7. 2016Language to Logical Form with Neural Attentioncited by 158
  8. 2016Data Recombination for Neural Semantic Parsingcited by 111
  9. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  10. 2020Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspectivecited by 95
  11. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  12. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  13. 2021An Improved Baseline for Sentence-level Relation Extractioncited by 81
  14. 2017Recurrent Relational Networkscited by 72
  15. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
  16. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  17. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 68
  18. 2020Learning Reasoning Strategies in End-to-End Differentiable Provingcited by 60
  19. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  20. 2021Neuro-Symbolic Artificial Intelligence: Current Trendscited by 54
  21. 2021Systematic Generalization with Edge Transformerscited by 38
  22. 2022Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoningcited by 34
  23. 2020Self-Attentive Associative Memorycited by 33
  24. 2018Learning to Reason with Third-Order Tensor Productscited by 27
  25. 2021The Power of Prompt Tuning for Low-Resource Semantic Parsingcited by 21
  26. 2021Systematic Generalization on gSCAN: What is Nearly Solved and What is Next?cited by 15
  27. 2017Weakly-supervised Semantic Parsing with Abstract Examplescited by 13
  28. 2016Semantic Parsing with Semi-Supervised Sequential Autoencoderscited by 11
  29. 2018Weight Learning in a Probabilistic Extension of Answer Set Programscited by 4
  30. 2014Reformulating the Situation Calculus and the Event Calculus in the General Theory of Stable Models and in Answer Set Programmingcited by 3
  31. 2022IELM: An Open Information Extraction Benchmark for Pre-Trained Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 18; 2026: 13.

59 papers.

Sort:

By area:

  1. 2024Understanding the planning of LLM agents: A surveycited by 421
  2. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 66
  3. 2023Dynamic Planning with a LLMcited by 61
  4. 2024TopViewRS: Vision-Language Models as Top-View Spatial Reasonerscited by 49
  5. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  6. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  7. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 34
  8. 2024Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languagescited by 33
  9. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 32
  10. 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmarkcited by 27
  11. 2024Planning with Multi-Constraints via Collaborative Language Agentscited by 26
  12. 2025NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domainscited by 15
38 more
  1. 2024BlendRL: A Framework for Merging Symbolic and Neural Policy Learningcited by 14
  2. 2025HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinkingcited by 14
  3. 2024The Role of Foundation Models in Neuro-Symbolic Learning and Reasoningcited by 12
  4. 2024A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Modelscited by 12
  5. 2025VERUS-LM: a Versatile Framework for Combining LLMs with Symbolic Reasoningcited by 12
  6. 2025SOP-Agent: Empower General Purpose AI Agent with Domain-Specific SOPscited by 11
  7. 2025Synergizing Logical Reasoning, Knowledge Management and Collaboration in Multi-Agent LLM Systemcited by 10
  8. 2024DeiSAM: Segment Anything with Deictic Promptingcited by 9
  9. 2025A Neurosymbolic Approach to Natural Language Formalization and Verificationcited by 9
  10. 2024LLASP: Fine-tuning Large Language Models for Answer Set Programmingcited by 8
  11. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  12. 2024Towards Automatic Composition of ASP Programs from Natural Language Specificationscited by 7
  13. 2024ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoningcited by 7
  14. 2025Towards a Common Framework for Autoformalizationcited by 7
  15. 2024Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMscited by 6
  16. 2025Adaptive Domain Modeling with Language Models: A Multi-Agent Approach to Task Planningcited by 6
  17. 2024SpaRC and SpaRP: Spatial Reasoning Characterization and Path Generation for Understanding Spatial Reasoning Capability of Large Language Modelscited by 5
  18. 2024Towards Logically Sound Natural Language Reasoning with Logic-Enhanced Language Model Agentscited by 5
  19. 2024Autoformalization of Game Descriptions using Large Language Modelscited by 5
  20. 2024Generative Agents for Multi-Agent Autoformalization of Interaction Scenarioscited by 5
  21. 2025NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoningcited by 5
  22. 2026Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systemscited by 5
  23. 2025Building a Stable Planner: An Extended Finite State Machine Based Planning Module for Mobile GUI Agentcited by 4
  24. 2025A Comparative Study of Neurosymbolic AI Approaches to Interpretable Logical Reasoningcited by 4
  25. 2025Question Answering with LLMs and Learning from Answer Setscited by 4
  26. 2026STAR: Mitigating Cascading Errors in Spatial Reasoning via Turn-point Alignment and Segment-level DPOcited by 4
  27. 2024CLMASP: Coupling Large Language Models with Answer Set Programming for Robotic Task Planningcited by 3
  28. 2024Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Languagecited by 3
  29. 2025An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoningcited by 3
  30. 2025Cooking Up Creativity: Enhancing LLM Creativity through Structured Recombinationcited by 3
  31. 2026LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Modelscited by 3
  32. 2024TIC: Translate-Infer-Compile for accurate "text to plan" using LLMs and Logical Representationscited by 2
  33. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  34. 2024ACCORD: Closing the Commonsense Measurability Gapcited by 1
  35. 2024A Training Data Recipe to Accelerate A* Search with Language Modelscited by 1
  36. 2024Effective Large Language Model Debugging with Best-first Tree Searchcited by 1
  37. 2026Spatial Reasoning via Modality Switching Between Language and Symbolic Representationscited by 1
  38. 2025LLMSR@XLLM25: An Empirical Study of LLM for Structural Reasoningcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Leveraging Large Language Models to Generate Answer Set Programs
  2. 2025Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning
  3. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  4. 2023Reliable Natural Language Understanding with Large Language Models and Answer Set Programming
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
  6. 2026Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models
  7. 2023Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning
  8. 2025Logical Reasoning in Large Language Models: A Survey