# Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels

arXiv: https://arxiv.org/abs/2110.12536 · HTML: https://mlsanity.com/paper/2110.12536/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 48 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Do Convolutional Neural Networks Learn Class Hierarchy?](https://mlsanity.com/paper/1710.06501/) (2017)
- [Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels](https://mlsanity.com/paper/2101.04893/) (2021)
- [ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models](https://mlsanity.com/paper/1704.01942/) (2017)
- [Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models](https://mlsanity.com/paper/1808.00196/) (2018)
- [ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion](https://mlsanity.com/paper/1910.00969/) (2019)
- [mage: Fluid Moves Between Code and Graphical Work in Computational Notebooks](https://mlsanity.com/paper/2009.10643/) (2020)
- [Classifier uncertainty: evidence, potential impact, and probabilistic treatment](https://mlsanity.com/paper/2006.11105/) (2020)
- [Visualizing Classification Structure of Large-Scale Classifiers](https://mlsanity.com/paper/2007.06068/) (2020)

## Built on by (most cited first, 17 of 17)

- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences](https://mlsanity.com/paper/2310.04621/) (2023)
- [A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision](https://mlsanity.com/paper/2308.05168/) (2023)
- [Calibrate: Interactive Analysis of Probabilistic Model Output](https://mlsanity.com/paper/2207.13770/) (2022)
- [Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference](https://mlsanity.com/paper/2404.03085/) (2024)
- [ParaDime: A Framework for Parametric Dimensionality Reduction](https://mlsanity.com/paper/2210.04582/) (2022)
- [Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods](https://mlsanity.com/paper/2508.12730/) (2025)
- [Designing Data: Proactive Data Collection and Iteration for Machine Learning](https://mlsanity.com/paper/2301.10319/) (2023)
- [Enhancing Bangla Fake News Detection Using Bidirectional Gated Recurrent Units and Deep Learning Techniques](https://mlsanity.com/paper/2404.01345/) (2024)
- [RuleExplorer: A Scalable Matrix Visualization for Understanding Tree Ensemble Classifiers](https://mlsanity.com/paper/2409.03164/) (2024)
- [ChannelExplorer: Exploring Class Separability Through Activation Channel Visualization](https://mlsanity.com/paper/2505.04647/) (2025)
- [BI-LAVA: Biocuration with Hierarchical Image Labeling through Active Learning and Visual Analysis](https://mlsanity.com/paper/2308.08003/) (2023)
- [Circles: Inter-Model Comparison of Multi-Classification Problems with High Number of Classes](https://mlsanity.com/paper/2309.05672/) (2023)
- [Diabetic Retinopathy Detection Based on Convolutional Neural Networks with SMOTE and CLAHE Techniques Applied to Fundus Images](https://mlsanity.com/paper/2504.05696/) (2025)
- [On the Normalization of Confusion Matrices: Methods and Geometric Interpretations](https://mlsanity.com/paper/2509.04959/) (2025)
- [An Energy-Efficient Smart Bus Transport Management System with Blind-Spot Collision Detection Ability](https://mlsanity.com/paper/2601.01274/) (2026)

## Abstract (arXiv)

The confusion matrix, a ubiquitous visualization for helping people evaluate machine learning models, is a tabular layout that compares predicted class labels against actual class labels over all data instances. We conduct formative research with machine learning practitioners at Apple and find that conventional confusion matrices do not support more complex data-structures found in modern-day applications, such as hierarchical and multi-output labels. To express such variations of confusion matrices, we design an algebra that models confusion matrices as probability distributions. Based on this algebra, we develop Neo, a visual analytics system that enables practitioners to flexibly author and interact with hierarchical and multi-output confusion matrices, visualize derived metrics, renormalize confusions, and share matrix specifications. Finally, we demonstrate Neo's utility with three model evaluation scenarios that help people better understand model performance and reveal hidden confusions.