# A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision

arXiv: https://arxiv.org/abs/2308.05168 · HTML: https://mlsanity.com/paper/2308.05168/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 74 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images](https://mlsanity.com/paper/1905.12886/) (2019)
- [Boundary loss for highly unbalanced segmentation](https://mlsanity.com/paper/1812.07032/) (2018)
- [Focal Modulation Networks](https://mlsanity.com/paper/2203.11926/) (2022)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [TIDE: A General Toolbox for Identifying Object Detection Errors](https://mlsanity.com/paper/2008.08115/) (2020)
- [Do Convolutional Neural Networks Learn Class Hierarchy?](https://mlsanity.com/paper/1710.06501/) (2017)
- [A Survey of Visual Analytics Techniques for Machine Learning](https://mlsanity.com/paper/2008.09632/) (2020)
- [Towards Better Analysis of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.07043/) (2016)
- [Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models](https://mlsanity.com/paper/2009.07053/) (2020)
- [Geometric Understanding of Deep Learning](https://mlsanity.com/paper/1805.10451/) (2018)
- [OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples](https://mlsanity.com/paper/2002.03103/) (2020)
- [Analyzing the Noise Robustness of Deep Neural Networks](https://mlsanity.com/paper/2001.09395/) (2020)
- [Boxer: Interactive Comparison of Classifier Results](https://mlsanity.com/paper/2004.07964/) (2020)
- [VATLD: A Visual Analytics System to Assess, Understand and Improve Traffic Light Detection](https://mlsanity.com/paper/2009.12975/) (2020)
- [A Unified Understanding of Deep NLP Models for Text Classification](https://mlsanity.com/paper/2206.09355/) (2022)
- [Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels](https://mlsanity.com/paper/2110.12536/) (2021)

## Built on by (most cited first, 8 of 8)

- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets](https://mlsanity.com/paper/2412.16839/) (2024)
- [AttributionScanner: A Visual Analytics System for Model Validation with Metadata-Free Slice Finding](https://mlsanity.com/paper/2401.06462/) (2024)
- [VISLIX: An XAI Framework for Validating Vision Models with Slice Discovery and Analysis](https://mlsanity.com/paper/2505.03132/) (2025)
- [Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods](https://mlsanity.com/paper/2508.12730/) (2025)
- [RuleExplorer: A Scalable Matrix Visualization for Understanding Tree Ensemble Classifiers](https://mlsanity.com/paper/2409.03164/) (2024)
- [VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models](https://mlsanity.com/paper/2509.14571/) (2025)
- [Survey on Remote Sensing Scene Classification: From Traditional Methods to Large Generative AI Models](https://mlsanity.com/paper/2603.26751/) (2026)

## Abstract (arXiv)

Existing model evaluation tools mainly focus on evaluating classification models, leaving a gap in evaluating more complex models, such as object detection. In this paper, we develop an open-source visual analysis tool, Uni-Evaluator, to support a unified model evaluation for classification, object detection, and instance segmentation in computer vision. The key idea behind our method is to formulate both discrete and continuous predictions in different tasks as unified probability distributions. Based on these distributions, we develop 1) a matrix-based visualization to provide an overview of model performance; 2) a table visualization to identify the problematic data subsets where the model performs poorly; 3) a grid visualization to display the samples of interest. These visualizations work together to facilitate the model evaluation from a global overview to individual samples. Two case studies demonstrate the effectiveness of Uni-Evaluator in evaluating model performance and making informed improvements.