# Question Answering with LLMs and Learning from Answer Sets

arXiv: https://arxiv.org/abs/2509.16590 · HTML: https://mlsanity.com/paper/2509.16590/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 19 of its 43 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [Multi-shot ASP solving with clingo](https://mlsanity.com/paper/1705.09811/) (2017)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Inductive logic programming at 30: a new introduction](https://mlsanity.com/paper/2008.07912/) (2020)
- [Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2107.02794/) (2021)
- [ASP-Core-2 Input Language Format](https://mlsanity.com/paper/1911.04326/) (2019)
- [Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text](https://mlsanity.com/paper/2307.07696/) (2023)
- [Word meaning in minds and machines](https://mlsanity.com/paper/2008.01766/) (2020)
- [Empowering LLMs with Logical Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2502.15652/) (2025)
- [Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.00836/) (2023)
- [Leveraging Large Language Models to Generate Answer Set Programs](https://mlsanity.com/paper/2307.07699/) (2023)
- [The ILASP system for Inductive Learning of Answer Set Programs](https://mlsanity.com/paper/2005.00904/) (2020)
- [Numerical reasoning in machine reading comprehension tasks: are we there yet?](https://mlsanity.com/paper/2109.08207/) (2021)
- [Learning to Correct for QA Reasoning with Black-box LLMs](https://mlsanity.com/paper/2406.18695/) (2024)
- [Incremental Learning of Event Definitions with Inductive Logic Programming](https://mlsanity.com/paper/1402.5988/) (2014)

## Built on by (most cited first, 4 of 4)

- [Scaling Reasoning Efficiently via Relaxed On-Policy Distillation](https://mlsanity.com/paper/2603.11137/) (2026)
- [Bridging Natural Language and ASP: A Hybrid Approach Using LLMs and AMR Parsing](https://mlsanity.com/paper/2511.08715/) (2025)
- [Modeling Next-Token Prediction as Left-Nested Intuitionistic Implication](https://mlsanity.com/paper/2601.19915/) (2026)
- [Distilling Answer Set Programming Theories from Large Language Models](https://mlsanity.com/paper/2607.28086/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) excel at understanding natural language but struggle with explicit commonsense reasoning. A recent trend of research suggests that the combination of LLM with robust symbolic reasoning systems can overcome this problem on story-based question answering tasks. In this setting, existing approaches typically depend on human expertise to manually craft the symbolic component. We argue, however, that this component can also be automatically learned from examples. In this work, we introduce LLM2LAS, a hybrid system that effectively combines the natural language understanding capabilities of LLMs, the rule induction power of the Learning from Answer Sets (LAS) system ILASP, and the formal reasoning strengths of Answer Set Programming (ASP). LLMs are used to extract semantic structures from text, which ILASP then transforms into interpretable logic rules. These rules allow an ASP solver to perform precise and consistent reasoning, enabling correct answers to previously unseen questions. Empirical results outline the strengths and weaknesses of our automatic approach for learning and reasoning in a story-based question answering benchmark.