# Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasets

arXiv: https://arxiv.org/abs/2410.09428 · HTML: https://mlsanity.com/paper/2410.09428/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)
- [ASP-Core-2 Input Language Format](https://mlsanity.com/paper/1911.04326/) (2019)
- [Large Language Models can Learn Rules](https://mlsanity.com/paper/2310.07064/) (2023)
- [The ILASP system for Inductive Learning of Answer Set Programs](https://mlsanity.com/paper/2005.00904/) (2020)
- [LRTA: A Transparent Neural-Symbolic Reasoning Framework with Modular Supervision for Visual Question Answering](https://mlsanity.com/paper/2011.10731/) (2020)
- [GPT4All: An Ecosystem of Open Source Compressed Language Models](https://mlsanity.com/paper/2311.04931/) (2023)

## Built on by (most cited first, 1 of 1)

- [Distilling Answer Set Programming Theories from Large Language Models](https://mlsanity.com/paper/2607.28086/) (2026)

## Abstract (arXiv)

Visual Question Answering (VQA) is the task of answering a question about an image and requires processing multimodal input and reasoning to obtain the answer. Modular solutions that use declarative representations within the reasoning component have a clear advantage over end-to-end trained systems regarding interpretability. The downside is that crafting the rules for such a component can be an additional burden on the developer. We address this challenge by presenting an approach for declarative knowledge distillation from Large Language Models (LLMs). Our method is to prompt an LLM to extend an initial theory on VQA reasoning, given as an answer-set program, to meet the requirements of the VQA task. Examples from the VQA dataset are used to guide the LLM, validate the results, and mend rules if they are not correct by using feedback from the ASP solver. We demonstrate that our approach works on the prominent CLEVR and GQA datasets. Our results confirm that distilling knowledge from LLMs is in fact a promising direction besides data-driven rule learning approaches.