# Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure

arXiv: https://arxiv.org/abs/1809.07424 · HTML: https://mlsanity.com/paper/1809.07424/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2018.

## Built on by (most cited first, 30 of 49)

- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [Salient ImageNet: How to discover spurious features in Deep Learning?](https://mlsanity.com/paper/2110.04301/) (2021)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [A Multimodal Automated Interpretability Agent](https://mlsanity.com/paper/2404.14394/) (2024)
- [Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs](https://mlsanity.com/paper/2103.06076/) (2021)
- [Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning](https://mlsanity.com/paper/2304.03916/) (2023)
- [ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO](https://mlsanity.com/paper/2204.03359/) (2022)
- [Improving Human-AI Collaboration With Descriptions of AI Behavior](https://mlsanity.com/paper/2301.06937/) (2023)
- [Mass-Producing Failures of Multimodal Systems with Language Models](https://mlsanity.com/paper/2306.12105/) (2023)
- [MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs](https://mlsanity.com/paper/2406.17126/) (2024)
- [Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice](https://mlsanity.com/paper/2210.03709/) (2022)
- [Discovering and Validating AI Errors With Crowdsourced Failure Reports](https://mlsanity.com/paper/2109.11690/) (2021)
- [Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment](https://mlsanity.com/paper/2202.02838/) (2022)
- [Core Risk Minimization using Salient ImageNet](https://mlsanity.com/paper/2203.15566/) (2022)
- [Establishing and Evaluating Trustworthy AI: Overview and Research Challenges](https://mlsanity.com/paper/2411.09973/) (2024)
- [Invariant Learning via Diffusion Dreamed Distribution Shifts](https://mlsanity.com/paper/2211.10370/) (2022)
- [fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks](https://mlsanity.com/paper/2302.11703/) (2023)
- [Learning Robust Classifiers with Self-Guided Spurious Correlation Mitigation](https://mlsanity.com/paper/2405.03649/) (2024)
- [PRIME: Prioritizing Interpretability in Failure Mode Extraction](https://mlsanity.com/paper/2310.00164/) (2023)
- [Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models](https://mlsanity.com/paper/2410.13826/) (2024)
- [Human-AI Symbiosis: A Survey of Current Approaches](https://mlsanity.com/paper/2103.09990/) (2021)
- [Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows](https://mlsanity.com/paper/2201.07677/) (2022)
- [Predictable Artificial Intelligence](https://mlsanity.com/paper/2310.06167/) (2023)
- [Explaining CLIP's performance disparities on data from blind/low vision users](https://mlsanity.com/paper/2311.17315/) (2023)
- [Spuriousness-Aware Meta-Learning for Learning Robust Classifiers](https://mlsanity.com/paper/2406.10742/) (2024)
- [Understanding and Avoiding AI Failures: A Practical Guide](https://mlsanity.com/paper/2104.12582/) (2021)
- [Benchmarking Spurious Bias in Few-Shot Image Classifiers](https://mlsanity.com/paper/2409.02882/) (2024)
- [To Trust or Not to Trust a Regressor: Estimating and Explaining Trustworthiness of Regression Predictions](https://mlsanity.com/paper/2104.06982/) (2021)
- [An Empirical Exploration of Trust Dynamics in LLM Supply Chains](https://mlsanity.com/paper/2405.16310/) (2024)

## Abstract (arXiv)

As machine learning systems move from computer-science laboratories into the open world, their accountability becomes a high priority problem. Accountability requires deep understanding of system behavior and its failures. Current evaluation methods such as single-score error metrics and confusion matrices provide aggregate views of system performance that hide important shortcomings. Understanding details about failures is important for identifying pathways for refinement, communicating the reliability of systems in different settings, and for specifying appropriate human oversight and engagement. Characterization of failures and shortcomings is particularly complex for systems composed of multiple machine learned components. For such systems, existing evaluation methods have limited expressiveness in describing and explaining the relationship among input content, the internal states of system components, and final output quality. We present Pandora, a set of hybrid human-machine methods and tools for describing and explaining system failures. Pandora leverages both human and system-generated observations to summarize conditions of system malfunction with respect to the input content and system architecture. We share results of a case study with a machine learning pipeline for image captioning that show how detailed performance views can be beneficial for analysis and debugging.