# Efficient Failure Pattern Identification of Predictive Algorithms

arXiv: https://arxiv.org/abs/2306.00760 · HTML: https://mlsanity.com/paper/2306.00760/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020)
- [Geometric Dataset Distances via Optimal Transport](https://mlsanity.com/paper/2002.02923/) (2020)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1910.02923/) (2019)
- [Graph-based Semi-supervised Learning: A Comprehensive Review](https://mlsanity.com/paper/2102.13303/) (2021)
- [Automated Data Slicing for Model Validation:A Big data - AI Integration Approach](https://mlsanity.com/paper/1807.06068/) (2018)
- [The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models](https://mlsanity.com/paper/2107.00758/) (2021)
- [Understanding Failures of Deep Networks via Robust Feature Extraction](https://mlsanity.com/paper/2012.01750/) (2020)
- [Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure](https://mlsanity.com/paper/1809.07424/) (2018)
- [Learning the Parameters of Determinantal Point Process Kernels](https://mlsanity.com/paper/1402.4862/) (2014)
- [Kernel Methods on the Riemannian Manifold of Symmetric Positive Definite Matrices](https://mlsanity.com/paper/1412.4172/) (2014)
- [Accelerating Human-in-the-loop Machine Learning: Challenges and Opportunities](https://mlsanity.com/paper/1804.05892/) (2018)
- [Multi-Document Summarization with Determinantal Point Processes and Contextualized Representations](https://mlsanity.com/paper/1910.11411/) (2019)
- [Low-Rank Factorization of Determinantal Point Processes for Recommendation](https://mlsanity.com/paper/1602.05436/) (2016)
- [Dynamic Flows on Curved Space Generated by Labeled Data](https://mlsanity.com/paper/2302.00061/) (2023)

## Abstract (arXiv)

Given a (machine learning) classifier and a collection of unlabeled data, how can we efficiently identify misclassification patterns presented in this dataset? To address this problem, we propose a human-machine collaborative framework that consists of a team of human annotators and a sequential recommendation algorithm. The recommendation algorithm is conceptualized as a stochastic sampler that, in each round, queries the annotators a subset of samples for their true labels and obtains the feedback information on whether the samples are misclassified. The sampling mechanism needs to balance between discovering new patterns of misclassification (exploration) and confirming the potential patterns of classification (exploitation). We construct a determinantal point process, whose intensity balances the exploration-exploitation trade-off through the weighted update of the posterior at each round to form the generator of the stochastic sampler. The numerical results empirically demonstrate the competitive performance of our framework on multiple datasets at various signal-to-noise ratios.