# How Trustworthy are Performance Evaluations for Basic Vision Tasks?

arXiv: https://arxiv.org/abs/2008.03533 · HTML: https://mlsanity.com/paper/2008.03533/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2020.

## Built on by (most cited first, 14 of 14)

- [JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and Tracking](https://mlsanity.com/paper/2210.11940/) (2022)
- [Gaussian Bounding Boxes and Probabilistic Intersection-over-Union for Object Detection](https://mlsanity.com/paper/2106.06072/) (2021)
- [JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environments](https://mlsanity.com/paper/2404.01686/) (2024)
- [Visual Multi-Object Tracking with Re-Identification and Occlusion Handling using Labeled Random Finite Sets](https://mlsanity.com/paper/2407.08872/) (2024)
- [Track Initialization and Re-Identification for~3D Multi-View Multi-Object Tracking](https://mlsanity.com/paper/2405.18606/) (2024)
- [JRDB-Traj: A Dataset and Benchmark for Trajectory Forecasting in Crowds](https://mlsanity.com/paper/2311.02736/) (2023)
- [Foundations of the Theory of Performance-Based Ranking](https://mlsanity.com/paper/2412.04227/) (2024)
- [The Tile: A 2D Map of Ranking Scores for Two-Class Classification](https://mlsanity.com/paper/2412.04309/) (2024)
- [TGOSPA Metric Parameters Selection and Evaluation for Visual Multi-object Tracking](https://mlsanity.com/paper/2412.08321/) (2024)
- [From Target Tracking to Targeting Track -- Part I: A Metric for Spatio-Temporal Trajectory Evaluation](https://mlsanity.com/paper/2502.15842/) (2025)
- [T-DuMpRa: Teacher-guided Dual-path Multi-prototype Retrieval Augmented framework for fine-grained medical image classification](https://mlsanity.com/paper/2604.17360/) (2026)
- [Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Driving](https://mlsanity.com/paper/2308.07106/) (2023)
- [Model-based Multi-object Visual Tracking: Identification and Standard Model Limitations](https://mlsanity.com/paper/2508.13647/) (2025)
- [End-to-end plaque counting and virus titration from laboratory plate images with deep learning](https://mlsanity.com/paper/2605.16008/) (2026)

## Abstract (arXiv)

This paper examines performance evaluation criteria for basic vision tasks involving sets of objects namely, object detection, instance-level segmentation and multi-object tracking. The rankings of algorithms by an existing criterion can fluctuate with different choices of parameters, e.g. Intersection over Union (IoU) threshold, making their evaluations unreliable. More importantly, there is no means to verify whether we can trust the evaluations of a criterion. This work suggests a notion of trustworthiness for performance criteria, which requires (i) robustness to parameters for reliability, (ii) contextual meaningfulness in sanity tests, and (iii) consistency with mathematical requirements such as the metric properties. We observe that these requirements were overlooked by many widely-used criteria, and explore alternative criteria using metrics for sets of shapes. We also assess all these criteria based on the suggested requirements for trustworthiness.