# Evaluating the Interpretability of Generative Models by Interactive Reconstruction

arXiv: https://arxiv.org/abs/2102.01264 · HTML: https://mlsanity.com/paper/2102.01264/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 78 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making](https://mlsanity.com/paper/1902.02960/) (2019)
- [An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1902.00006/) (2019)
- [What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play](https://mlsanity.com/paper/1810.09648/) (2018)
- [Embedding Projector: Interactive Visualization and Interpretation of Embeddings](https://mlsanity.com/paper/1611.05469/) (2016)
- [A Loss Function for Generative Neural Networks Based on Watson's Perceptual Model](https://mlsanity.com/paper/2006.15057/) (2020)
- [Human-in-the-Loop Interpretability Prior](https://mlsanity.com/paper/1805.11571/) (2018)
- [TensorFlow.js: Machine Learning for the Web and Beyond](https://mlsanity.com/paper/1901.05350/) (2019)
- [A Survey of Inductive Biases for Factorial Representation-Learning](https://mlsanity.com/paper/1612.05299/) (2016)
- [Assessing the Local Interpretability of Machine Learning Models](https://mlsanity.com/paper/1902.03501/) (2019)
- [A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration](https://mlsanity.com/paper/1811.12199/) (2018)

## Built on by (most cited first, 17 of 17)

- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Investigating Explainability of Generative AI for Code through Scenario-based Design](https://mlsanity.com/paper/2202.04903/) (2022)
- [Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents](https://mlsanity.com/paper/2308.01542/) (2023)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [Interactive Disentanglement: Learning Concepts by Interacting with their Prototype Representations](https://mlsanity.com/paper/2112.02290/) (2021)
- [Understanding Instance-based Interpretability of Variational Auto-Encoders](https://mlsanity.com/paper/2105.14203/) (2021)
- [When are Post-hoc Conceptual Explanations Identifiable?](https://mlsanity.com/paper/2206.13872/) (2022)
- [The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?](https://mlsanity.com/paper/2410.04699/) (2024)
- [GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information](https://mlsanity.com/paper/2202.00965/) (2022)
- [Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglement](https://mlsanity.com/paper/2102.05185/) (2021)
- [Generative AI in Writing Research Papers: A New Type of Algorithmic Bias and Uncertainty in Scholarly Work](https://mlsanity.com/paper/2312.10057/) (2023)
- [Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces](https://mlsanity.com/paper/2111.14951/) (2021)
- [Latent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration](https://mlsanity.com/paper/2312.00857/) (2023)
- [A Systematic Comparison between Extractive Self-Explanations and Human Rationales in Text Classification](https://mlsanity.com/paper/2410.03296/) (2024)
- [Measuring algorithmic interpretability: A human-learning-based framework and the corresponding cognitive complexity score](https://mlsanity.com/paper/2205.10207/) (2022)
- [Selecting Interpretability Techniques for Healthcare Machine Learning models](https://mlsanity.com/paper/2406.10213/) (2024)
- [Attribution assignment for deep-generative sequence models enables interpretability analysis using positive-only data](https://mlsanity.com/paper/2506.23182/) (2025)

## Abstract (arXiv)

For machine learning models to be most useful in numerous sociotechnical systems, many have argued that they must be human-interpretable. However, despite increasing interest in interpretability, there remains no firm consensus on how to measure it. This is especially true in representation learning, where interpretability research has focused on "disentanglement" measures only applicable to synthetic datasets and not grounded in human factors. We introduce a task to quantify the human-interpretability of generative model representations, where users interactively modify representations to reconstruct target instances. On synthetic datasets, we find performance on this task much more reliably differentiates entangled and disentangled models than baseline approaches. On a real dataset, we find it differentiates between representation learning methods widely believed but never shown to produce more or less interpretable models. In both cases, we ran small-scale think-aloud studies and large-scale experiments on Amazon Mechanical Turk to confirm that our qualitative and quantitative results agreed.