# Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning

arXiv: https://arxiv.org/abs/1803.05268 · HTML: https://mlsanity.com/paper/1803.05268/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2018.

## Built on by (most cited first, 30 of 51)

- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond](https://mlsanity.com/paper/2102.01998/) (2021)
- [Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning](https://mlsanity.com/paper/2103.16564/) (2021)
- [NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations](https://mlsanity.com/paper/2303.13483/) (2023)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)
- [Few-shot Backdoor Defense Using Shapley Estimation](https://mlsanity.com/paper/2112.14889/) (2021)
- [Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing](https://mlsanity.com/paper/2210.15889/) (2022)
- [What's Left? Concept Grounding with Logic-Enhanced Foundation Models](https://mlsanity.com/paper/2310.16035/) (2023)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?](https://mlsanity.com/paper/2304.04780/) (2023)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [Neuro-Symbolic AI: An Emerging Class of AI Workloads and their Characterization](https://mlsanity.com/paper/2109.06133/) (2021)
- [FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations](https://mlsanity.com/paper/2203.16639/) (2022)
- [Towards Language-guided Visual Recognition via Dynamic Convolutions](https://mlsanity.com/paper/2110.08797/) (2021)
- [Learn-Explain-Reinforce: Counterfactual Reasoning and Its Guidance to Reinforce an Alzheimer's Disease Diagnosis Model](https://mlsanity.com/paper/2108.09451/) (2021)
- [Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images](https://mlsanity.com/paper/2110.00519/) (2021)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines](https://mlsanity.com/paper/2502.16641/) (2025)
- [Hopper: Multi-hop Transformer for Spatiotemporal Reasoning](https://mlsanity.com/paper/2103.10574/) (2021)
- [Domain-robust VQA with diverse datasets and methods but no target labels](https://mlsanity.com/paper/2103.15974/) (2021)
- [Grammar-Based Grounded Lexicon Learning](https://mlsanity.com/paper/2202.08806/) (2022)
- [From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?](https://mlsanity.com/paper/2403.11894/) (2024)
- [Solving the Same-Different Task with Convolutional Neural Networks](https://mlsanity.com/paper/2101.09129/) (2021)
- [Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inference](https://mlsanity.com/paper/2203.04857/) (2022)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Transformer Module Networks for Systematic Generalization in Visual Question Answering](https://mlsanity.com/paper/2201.11316/) (2022)
- [Neuro-Symbolic VQA: A review from the perspective of AGI desiderata](https://mlsanity.com/paper/2104.06365/) (2021)

## Abstract (arXiv)

Visual question answering requires high-order reasoning about an image, which is a fundamental capability needed by machine systems to follow complex directives. Recently, modular networks have been shown to be an effective framework for performing visual reasoning tasks. While modular networks were initially designed with a degree of model transparency, their performance on complex visual reasoning benchmarks was lacking. Current state-of-the-art approaches do not provide an effective mechanism for understanding the reasoning process. In this paper, we close the performance gap between interpretable models and state-of-the-art visual reasoning methods. We propose a set of visual-reasoning primitives which, when composed, manifest as a model capable of performing complex reasoning tasks in an explicitly-interpretable manner. The fidelity and interpretability of the primitives' outputs enable an unparalleled ability to diagnose the strengths and weaknesses of the resulting model. Critically, we show that these primitives are highly performant, achieving state-of-the-art accuracy of 99.1% on the CLEVR dataset. We also show that our model is able to effectively learn generalized representations when provided a small amount of data containing novel object attributes. Using the CoGenT generalization task, we show more than a 20 percentage point improvement over the current state of the art.