# Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models

arXiv: https://arxiv.org/abs/2108.03738 · HTML: https://mlsanity.com/paper/2108.03738/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2021.

## Built on by (most cited first, 17 of 17)

- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation](https://mlsanity.com/paper/2403.12075/) (2024)
- [DendroMap: Visual Exploration of Large-Scale Image Datasets for Machine Learning with Treemaps](https://mlsanity.com/paper/2205.06935/) (2022)
- [VLSlice: Interactive Vision-and-Language Slice Discovery](https://mlsanity.com/paper/2309.06703/) (2023)
- [ProtoExplorer: Interpretable Forensic Analysis of Deepfake Videos using Prototype Exploration and Refinement](https://mlsanity.com/paper/2309.11155/) (2023)
- [ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale](https://mlsanity.com/paper/2404.02990/) (2024)
- [Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning](https://mlsanity.com/paper/2311.01016/) (2023)
- [Learning New Concepts, Remembering the Old: Continual Learning for Multimodal Concept Bottleneck Models](https://mlsanity.com/paper/2411.17471/) (2024)
- [Latent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration](https://mlsanity.com/paper/2312.00857/) (2023)
- [Interactive Image Selection and Training for Brain Tumor Segmentation Network](https://mlsanity.com/paper/2406.03225/) (2024)
- [ESCAPE: Countering Systematic Errors from Machine's Blind Spots via Interactive Visual Analysis](https://mlsanity.com/paper/2303.09657/) (2023)
- [An Interactive Interpretability System for Breast Cancer Screening with Deep Learning](https://mlsanity.com/paper/2210.08979/) (2022)
- [InterVLS: Interactive Model Understanding and Improvement with Vision-Language Surrogates](https://mlsanity.com/paper/2311.03547/) (2023)
- [ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models](https://mlsanity.com/paper/2509.20376/) (2025)
- [InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillation](https://mlsanity.com/paper/2406.17838/) (2024)
- [SketchXplain: Intuitive Visual Explanations of Image Classifiers with Sketches](https://mlsanity.com/paper/2606.17646/) (2026)
- [ASAP: Visual Analytics for Identifying and Analyzing Image Patterns in AI-generated Images](https://mlsanity.com/paper/2609.27371/) (2026)

## Abstract (arXiv)

The interpretation of deep neural networks (DNNs) has become a key topic as more and more people apply them to solve various problems and making critical decisions. Concept-based explanations have recently become a popular approach for post-hoc interpretation of DNNs. However, identifying human-understandable visual concepts that affect model decisions is a challenging task that is not easily addressed with automatic approaches. We present a novel human-in-the-loop approach to generate user-defined concepts for model interpretation and diagnostics. Central to our proposal is the use of active learning, where human knowledge and feedback are combined to train a concept extractor with very little human labeling effort. We integrate this process into an interactive system, ConceptExtract. Through two case studies, we show how our approach helps analyze model behavior and extract human-friendly concepts for different machine learning tasks and datasets and how to use these concepts to understand the predictions, compare model performance and make suggestions for model refinement. Quantitative experiments show that our active learning approach can accurately extract meaningful visual concepts. More importantly, by identifying visual concepts that negatively affect model performance, we develop the corresponding data augmentation strategy that consistently improves model performance.