# ESCAPE: Countering Systematic Errors from Machine's Blind Spots via Interactive Visual Analysis

arXiv: https://arxiv.org/abs/2303.09657 · HTML: https://mlsanity.com/paper/2303.09657/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 64 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Concept Whitening for Interpretable Image Recognition](https://mlsanity.com/paper/2002.01650/) (2020)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019)
- [The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification](https://mlsanity.com/paper/1503.01161/) (2015)
- [The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models](https://mlsanity.com/paper/2107.00758/) (2021)
- [FairSight: Visual Analytics for Fairness in Decision Making](https://mlsanity.com/paper/1908.00176/) (2019)
- [Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models](https://mlsanity.com/paper/1808.00196/) (2018)
- [OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples](https://mlsanity.com/paper/2002.03103/) (2020)
- [Discovering and Validating AI Errors With Crowdsourced Failure Reports](https://mlsanity.com/paper/2109.11690/) (2021)
- [Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration](https://mlsanity.com/paper/1610.09064/) (2016)
- [A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations](https://mlsanity.com/paper/1705.01968/) (2017)
- [Investigating Human + Machine Complementarity for Recidivism Predictions](https://mlsanity.com/paper/1808.09123/) (2018)
- [Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models](https://mlsanity.com/paper/2108.03738/) (2021)
- [SEAL : Interactive Tool for Systematic Error Analysis and Labeling](https://mlsanity.com/paper/2210.05839/) (2022)
- [Diagnosing Concept Drift with Visual Analytics](https://mlsanity.com/paper/2007.14372/) (2020)
- [ConceptExplorer: Visual Analysis of Concept Driftsin Multi-source Time-series Data](https://mlsanity.com/paper/2007.15272/) (2020)
- [NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks](https://mlsanity.com/paper/2108.12931/) (2021)
- [Improving Visualization Interpretation Using Counterfactuals](https://mlsanity.com/paper/2107.10309/) (2021)
- [DASH: Visual Analytics for Debiasing Image Classification via User-Driven Synthetic Data Augmentation](https://mlsanity.com/paper/2209.06357/) (2022)
- [Biased Embeddings from Wild Data: Measuring, Understanding and Removing](https://mlsanity.com/paper/1806.06301/) (2018)
- [Unknown Examples & Machine Learning Model Generalization](https://mlsanity.com/paper/1808.08294/) (2018)
- [Contrastive Identification of Covariate Shift in Image Data](https://mlsanity.com/paper/2108.08000/) (2021)

## Built on by (most cited first, 2 of 2)

- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)
- [VISLIX: An XAI Framework for Validating Vision Models with Slice Discovery and Analysis](https://mlsanity.com/paper/2505.03132/) (2025)

## Abstract (arXiv)

Classification models learn to generalize the associations between data samples and their target classes. However, researchers have increasingly observed that machine learning practice easily leads to systematic errors in AI applications, a phenomenon referred to as AI blindspots. Such blindspots arise when a model is trained with training samples (e.g., cat/dog classification) where important patterns (e.g., black cats) are missing or periphery/undesirable patterns (e.g., dogs with grass background) are misleading towards a certain class. Even more sophisticated techniques cannot guarantee to capture, reason about, and prevent the spurious associations. In this work, we propose ESCAPE, a visual analytic system that promotes a human-in-the-loop workflow for countering systematic errors. By allowing human users to easily inspect spurious associations, the system facilitates users to spontaneously recognize concepts associated misclassifications and evaluate mitigation strategies that can reduce biased associations. We also propose two statistical approaches, relative concept association to better quantify the associations between a concept and instances, and debias method to mitigate spurious associations. We demonstrate the utility of our proposed ESCAPE system and statistical measures through extensive evaluation including quantitative experiments, usage scenarios, expert interviews, and controlled user experiments.