# Reliable Natural Language Understanding with Large Language Models and Answer Set Programming

arXiv: https://arxiv.org/abs/2302.03780 · HTML: https://mlsanity.com/paper/2302.03780/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Logical Neural Networks](https://mlsanity.com/paper/2006.13155/) (2020)
- [Few-Shot Semantic Parsing with Language Models Trained On Code](https://mlsanity.com/paper/2112.08696/) (2021)
- [QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships](https://mlsanity.com/paper/1811.08048/) (2018)
- [Constraint Answer Set Programming without Grounding](https://mlsanity.com/paper/1804.11162/) (2018)
- [Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review](https://mlsanity.com/paper/2202.12205/) (2022)
- [Justifications for Goal-Directed Constraint Answer Set Programming](https://mlsanity.com/paper/2009.10238/) (2020)
- [AUTO-DISCERN: Autonomous Driving Using Common Sense Reasoning](https://mlsanity.com/paper/2110.13606/) (2021)
- [A Physician Advisory System for Chronic Heart Failure Management Based on Knowledge Patterns](https://mlsanity.com/paper/1610.08115/) (2016)
- [Knowledge-driven Natural Language Understanding of English Text and its Applications](https://mlsanity.com/paper/2101.11707/) (2021)
- [CASPR: A Commonsense Reasoning-based Conversational Socialbot](https://mlsanity.com/paper/2110.05387/) (2021)
- [RestKB: A Library of Commonsense Knowledge about Dining at a Restaurant](https://mlsanity.com/paper/1909.08239/) (2019)

## Built on by (most cited first, 12 of 12)

- [A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verification](https://mlsanity.com/paper/2305.14752/) (2023)
- [Automated Interactive Domain-Specific Conversational Agents that Understand Human Dialogs](https://mlsanity.com/paper/2303.08941/) (2023)
- [Automated Theorem Provers Help Improve Large Language Model Reasoning](https://mlsanity.com/paper/2408.03492/) (2024)
- [On the use of neurosymbolic AI for defending against cyber attacks](https://mlsanity.com/paper/2408.04996/) (2024)
- [TIC: Translate-Infer-Compile for accurate "text to plan" using LLMs and Logical Representations](https://mlsanity.com/paper/2402.06608/) (2024)
- [Visual Graph Question Answering with ASP and LLMs for Language Parsing](https://mlsanity.com/paper/2502.09211/) (2025)
- [XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners](https://mlsanity.com/paper/2310.05502/) (2023)
- [Reliable Conversational Agents under ASP Control that Understand Natural Language](https://mlsanity.com/paper/2502.09237/) (2025)
- [Reliable Collaborative Conversational Agent System Based on LLMs and Answer Set Programming](https://mlsanity.com/paper/2505.06438/) (2025)
- [Bridging Natural Language and ASP: A Hybrid Approach Using LLMs and AMR Parsing](https://mlsanity.com/paper/2511.08715/) (2025)
- [An XAI View on Explainable ASP: Methods, Systems, and Perspectives](https://mlsanity.com/paper/2601.14764/) (2026)
- [The Dual Role of Abstracting over the Irrelevant in Symbolic Explanations: Cognitive Effort vs. Understanding](https://mlsanity.com/paper/2602.03467/) (2026)

## Abstract (arXiv)

Humans understand language by extracting information (meaning) from sentences, combining it with existing commonsense knowledge, and then performing reasoning to draw conclusions. While large language models (LLMs) such as GPT-3 and ChatGPT are able to leverage patterns in the text to solve a variety of NLP tasks, they fall short in problems that require reasoning. They also cannot reliably explain the answers generated for a given question. In order to emulate humans better, we propose STAR, a framework that combines LLMs with Answer Set Programming (ASP). We show how LLMs can be used to effectively extract knowledge -- represented as predicates -- from language. Goal-directed ASP is then employed to reliably reason over this knowledge. We apply the STAR framework to three different NLU tasks requiring reasoning: qualitative reasoning, mathematical reasoning, and goal-directed conversation. Our experiments reveal that STAR is able to bridge the gap of reasoning in NLU tasks, leading to significant performance improvements, especially for smaller LLMs, i.e., LLMs with a smaller number of parameters. NLU applications developed using the STAR framework are also explainable: along with the predicates generated, a justification in the form of a proof tree can be produced for a given output.