# Towards Visual Explainable Active Learning for Zero-Shot Classification

arXiv: https://arxiv.org/abs/2108.06730 · HTML: https://mlsanity.com/paper/2108.06730/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 79 references that are in the map.
- First posted on arXiv 15 Aug 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making](https://mlsanity.com/paper/1902.02960/) (2019)
- [Leveraging the Invariant Side of Generative Zero-Shot Learning](https://mlsanity.com/paper/1904.04092/) (2019)
- [RuleMatrix: Visualizing and Understanding Classifiers with Rules](https://mlsanity.com/paper/1807.06228/) (2018)
- [Towards Better Analysis of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.07043/) (2016)
- [Understanding Hidden Memories of Recurrent Neural Networks](https://mlsanity.com/paper/1710.10777/) (2017)
- [Machine Teaching: A New Paradigm for Building Machine Learning Systems](https://mlsanity.com/paper/1707.06742/) (2017)
- [Visual Analytics for Explainable Deep Learning](https://mlsanity.com/paper/1804.02527/) (2018)
- [Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning](https://mlsanity.com/paper/1605.08151/) (2016)
- [Clustrophile 2: Guided Visual Clustering Analysis](https://mlsanity.com/paper/1804.03048/) (2018)
- [Recent Research Advances on Interactive Machine Learning](https://mlsanity.com/paper/1811.04548/) (2018)
- [Scalable Semi-Supervised SVM via Triply Stochastic Gradients](https://mlsanity.com/paper/1907.11584/) (2019)
- [Visually Analyzing and Steering Zero Shot Learning](https://mlsanity.com/paper/2009.05254/) (2020)

## Built on by (most cited first, 10 of 10)

- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [Diagnosing Ensemble Few-Shot Classifiers](https://mlsanity.com/paper/2206.04372/) (2022)
- [Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets](https://mlsanity.com/paper/2412.16839/) (2024)
- [VideoPro: A Visual Analytics Approach for Interactive Video Programming](https://mlsanity.com/paper/2308.00401/) (2023)
- [ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale](https://mlsanity.com/paper/2404.02990/) (2024)
- [Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning](https://mlsanity.com/paper/2311.01016/) (2023)
- [Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research](https://mlsanity.com/paper/2308.06290/) (2023)
- [OW-CLIP: Data-Efficient Visual Supervision for Open-World Object Detection via Human-AI Collaboration](https://mlsanity.com/paper/2507.19870/) (2025)
- [What Types of Human-AI Teams Exist?](https://mlsanity.com/paper/2607.02198/) (2026)

## Abstract (arXiv)

Zero-shot classification is a promising paradigm to solve an applicable problem when the training classes and test classes are disjoint. Achieving this usually needs experts to externalize their domain knowledge by manually specifying a class-attribute matrix to define which classes have which attributes. Designing a suitable class-attribute matrix is the key to the subsequent procedure, but this design process is tedious and trial-and-error with no guidance. This paper proposes a visual explainable active learning approach with its design and implementation called semantic navigator to solve the above problems. This approach promotes human-AI teaming with four actions (ask, explain, recommend, respond) in each interaction loop. The machine asks contrastive questions to guide humans in the thinking process of attributes. A novel visualization called semantic map explains the current status of the machine. Therefore analysts can better understand why the machine misclassifies objects. Moreover, the machine recommends the labels of classes for each attribute to ease the labeling burden. Finally, humans can steer the model by modifying the labels interactively, and the machine adjusts its recommendations. The visual explainable active learning approach improves humans' efficiency of building zero-shot classification models interactively, compared with the method without guidance. We justify our results with user studies using the standard benchmarks for zero-shot classification.