# Post-hoc Models for Performance Estimation of Machine Learning Inference

arXiv: https://arxiv.org/abs/2110.02459 · HTML: https://mlsanity.com/paper/2110.02459/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [Multivariate Confidence Calibration for Object Detection](https://mlsanity.com/paper/2004.13546/) (2020)
- [Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities](https://mlsanity.com/paper/1811.00648/) (2018)
- [YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers](https://mlsanity.com/paper/1811.05588/) (2018)
- [Real-time Prediction of Segmentation Quality](https://mlsanity.com/paper/1806.06244/) (2018)
- [MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection](https://mlsanity.com/paper/2010.01695/) (2020)
- [SurveilEdge: Real-time Video Query based on Collaborative Cloud-Edge Deep Learning](https://mlsanity.com/paper/2001.01043/) (2020)
- [From Black-box to White-box: Examining Confidence Calibration under different Conditions](https://mlsanity.com/paper/2101.02971/) (2021)
- [On the Role of Dataset Quality and Heterogeneity in Model Confidence](https://mlsanity.com/paper/2002.09831/) (2020)
- [Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices](https://mlsanity.com/paper/1709.09503/) (2017)
- [Per-frame mAP Prediction for Continuous Performance Monitoring of Object Detection During Deployment](https://mlsanity.com/paper/2009.08650/) (2020)

## Built on by (most cited first, 3 of 3)

- [Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations](https://mlsanity.com/paper/2403.01172/) (2024)
- [Run-time Monitoring of 3D Object Detection in Automated Driving Systems Using Early Layer Neural Activation Patterns](https://mlsanity.com/paper/2404.07685/) (2024)
- [RedunCut: Measurement-Driven Sampling and Accuracy Performance Modeling for Low-Cost Live Video Analytics](https://mlsanity.com/paper/2512.24386/) (2025)

## Abstract (arXiv)

Estimating how well a machine learning model performs during inference is critical in a variety of scenarios (for example, to quantify uncertainty, or to choose from a library of available models). However, the standard accuracy estimate of softmax confidence is not versatile and cannot reliably predict different performance metrics (e.g., F1-score, recall) or the performance in different application scenarios or input domains. In this work, we systematically generalize performance estimation to a diverse set of metrics and scenarios and discuss generalized notions of uncertainty calibration. We propose the use of post-hoc models to accomplish this goal and investigate design parameters, including the model type, feature engineering, and performance metric, to achieve the best estimation quality. Emphasis is given to object detection problems and, unlike prior work, our approach enables the estimation of per-image metrics such as recall and F1-score. Through extensive experiments with computer vision models and datasets in three use cases -- mobile edge offloading, model selection, and dataset shift -- we find that proposed post-hoc models consistently outperform the standard calibrated confidence baselines. To the best of our knowledge, this is the first work to develop a unified framework to address different performance estimation problems for machine learning inference.