# Progressive Interpretation Synthesis: Interpreting Task Solving by Quantifying Previously Used and Unused Information

arXiv: https://arxiv.org/abs/2101.02879 · HTML: https://mlsanity.com/paper/2101.02879/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 8 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Beyond Sparsity: Tree Regularization of Deep Models for Interpretability](https://mlsanity.com/paper/1711.06178/) (2017)
- [Interpreting CNN Knowledge via an Explanatory Graph](https://mlsanity.com/paper/1708.01785/) (2017)
- [The deterministic information bottleneck](https://mlsanity.com/paper/1604.00268/) (2016)
- [Explaining a black-box using Deep Variational Information Bottleneck Approach](https://mlsanity.com/paper/1902.06918/) (2019)
- [Specializing Word Embeddings (for Parsing) by Information Bottleneck](https://mlsanity.com/paper/1910.00163/) (2019)
- [Relevant sparse codes with variational information bottleneck](https://mlsanity.com/paper/1605.07332/) (2016)
- [Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions](https://mlsanity.com/paper/1510.00831/) (2015)
- [Visual Reasoning by Progressive Module Networks](https://mlsanity.com/paper/1806.02453/) (2018)

## Abstract (arXiv)

A deep neural network is a good task solver, but it is difficult to make sense of its operation. People have different ideas about how to form the interpretation about its operation. We look at this problem from a new perspective where the interpretation of task solving is synthesized by quantifying how much and what previously unused information is exploited in addition to the information used to solve previous tasks. First, after learning several tasks, the network acquires several information partitions related to each task. We propose that the network, then, learns the minimal information partition that supplements previously learned information partitions to more accurately represent the input. This extra partition is associated with un-conceptualized information that has not been used in previous tasks. We manage to identify what un-conceptualized information is used and quantify the amount. To interpret how the network solves a new task, we quantify as meta-information how much information from each partition is extracted. We implement this framework with the variational information bottleneck technique. We test the framework with the MNIST and the CLEVR dataset. The framework is shown to be able to compose information partitions and synthesize experience-dependent interpretation in the form of meta-information. This system progressively improves the resolution of interpretation upon new experience by converting a part of the un-conceptualized information partition to a task-related partition. It can also provide a visual interpretation by imaging what is the part of previously un-conceptualized information that is needed to solve a new task.