# Neuro-Symbolic Visual Dialog

arXiv: https://arxiv.org/abs/2208.10353 · HTML: https://mlsanity.com/paper/2208.10353/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 22 Aug 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [Task-Oriented Dialogue as Dataflow Synthesis](https://mlsanity.com/paper/2009.11423/) (2020)
- [Explicit Knowledge-based Reasoning for Visual Question Answering](https://mlsanity.com/paper/1511.02570/) (2015)
- [Visual Coreference Resolution in Visual Dialog using Neural Module Networks](https://mlsanity.com/paper/1809.01816/) (2018)
- [VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation](https://mlsanity.com/paper/1708.04686/) (2017)
- [CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog](https://mlsanity.com/paper/1903.03166/) (2019)
- [From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood](https://mlsanity.com/paper/1704.07926/) (2017)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Recursive Visual Attention in Visual Dialog](https://mlsanity.com/paper/1812.02664/) (2018)
- [Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning](https://mlsanity.com/paper/1702.03274/) (2017)
- [Conversational Neuro-Symbolic Commonsense Reasoning](https://mlsanity.com/paper/2006.10022/) (2020)
- [Reasoning Visual Dialogs with Structural and Partial Observations](https://mlsanity.com/paper/1904.05548/) (2019)
- [Learning to Map Context-Dependent Sentences to Executable Formal Queries](https://mlsanity.com/paper/1804.06868/) (2018)
- [Generating Descriptions with Grounded and Co-Referenced People](https://mlsanity.com/paper/1704.01518/) (2017)
- [Reasoning Over History: Context Aware Visual Dialog](https://mlsanity.com/paper/2011.00669/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Study on Neuro-Symbolic Artificial Intelligence: Healthcare Perspectives](https://mlsanity.com/paper/2503.18213/) (2025)
- [Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations](https://mlsanity.com/paper/2308.16349/) (2023)
- [V2Dial: Unification of Video and Visual Dialog via Multimodal Experts](https://mlsanity.com/paper/2503.02063/) (2025)

## Abstract (arXiv)

We propose Neuro-Symbolic Visual Dialog (NSVD) -the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog -a relative improvement of more than 10% over the state of the art while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.