Distilling Answer Set Programming Theories from Large Language Models
Not on the map. Why?
Abstract (arXiv)
Writing Answer Set Programming (ASP) theories from scratch is a difficult and time-consuming task. We take a neurosymbolic approach to study whether a model can distill complete and correct theories, given a fixed agent harness with the solver in the loop. The protocol is dataset-agnostic: with a single prompt and an empty file as the starting point the model is given a 1-hour time limit to derive a complete theory. We chose VQA as the application domain, three benchmarks (CLEVR, GQA, CLEVRER), as these are publicly available and non-trivial. In order to study the model scale required for solving this task we nine different models: four frontier (Claude Sonnet 4.6, Claude Opus 4.7, GPT-5, DeepSeek V4 Pro), two mid-tier (DeepSeek V4 Flash, gpt-oss-120b), and three open-weights (qwen3.6-27b, gpt-oss-20b, qwen3.5-9b). Three of four frontier models reach 100% on CLEVR and 92.8%-98.8% on GQA; on CLEVRER, Sonnet, Opus, DeepSeek V4 Pro score 92.7%-95.3%. GPT-5 reaches 98.7% on CLEVR but drops to 41.8% on GQA and to 86.7% on CLEVRER. Adding handwritten reference theories from other datasets moves the other three frontier models by at most +/-3.4 pp but reduces GPT-5's accuracy by 3-19 pp. We release the code, prompts, and theories distilled.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 30 Jul 2026.
Builds on
28 of 46 references.
References · 28
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2015VQA: Visual Question Answeringcited by 2,489
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
- 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
16 more
- 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 389
- 2015Neural Module Networkscited by 375
- 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
- 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
- 2020Neurosymbolic AI: The 3rd Wavecited by 253
- 2018Compositional Attention Networks for Machine Reasoningcited by 183
- 2017Inferring and Executing Programs for Visual Reasoningcited by 130
- 2017Multi-shot ASP solving with clingocited by 115
- 2017Logic Tensor Networks for Semantic Image Interpretationcited by 92
- 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
- 2023Leveraging Large Language Models to Generate Answer Set Programscited by 33
- 2022A Neuro-Symbolic ASP Pipeline for Visual Question Answeringcited by 14
- 2025Question Answering with LLMs and Learning from Answer Setscited by 4
- 2025A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solvingcited by 4
- 2024Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasetscited by 1
- 2025Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)cited by 1
Built on by
Not cited yet.