# Active Bayesian Assessment for Black-Box Classifiers

arXiv: https://arxiv.org/abs/2002.06532 · HTML: https://mlsanity.com/paper/2002.06532/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2020.

## Built on by (most cited first, 17 of 17)

- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.02666/) (2021)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation](https://mlsanity.com/paper/2202.06881/) (2022)
- [Label-Efficient Model Selection for Text Generation](https://mlsanity.com/paper/2402.07891/) (2024)
- [ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities](https://mlsanity.com/paper/2412.06745/) (2024)
- [Scaling Up Active Testing to Large Language Models](https://mlsanity.com/paper/2508.09093/) (2025)
- [Efficient Lifelong Model Evaluation in an Era of Rapid Progress](https://mlsanity.com/paper/2402.19472/) (2024)
- [Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks](https://mlsanity.com/paper/2501.04234/) (2025)
- [A structured regression approach for evaluating model performance across intersectional subgroups](https://mlsanity.com/paper/2401.14893/) (2024)
- [Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations](https://mlsanity.com/paper/2108.06514/) (2021)
- [DISCO: Diversifying Sample Condensation for Efficient Model Evaluation](https://mlsanity.com/paper/2510.07959/) (2025)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [Sequential Bayesian Evaluation of Large Language Model Behavior](https://mlsanity.com/paper/2511.10661/) (2025)
- [A Strong Baseline for Molecular Few-Shot Learning](https://mlsanity.com/paper/2404.02314/) (2024)
- [Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators](https://mlsanity.com/paper/2602.07226/) (2026)
- [Bayesian Learning to Select Fair Classifiers](https://mlsanity.com/paper/2610.20514/) (2026)

## Abstract (arXiv)

Recent advances in machine learning have led to increased deployment of black-box classifiers across a wide variety of applications. In many such situations there is a critical need to both reliably assess the performance of these pre-trained models and to perform this assessment in a label-efficient manner (given that labels may be scarce and costly to collect). In this paper, we introduce an active Bayesian approach for assessment of classifier performance to satisfy the desiderata of both reliability and label-efficiency. We begin by developing inference strategies to quantify uncertainty for common assessment metrics such as accuracy, misclassification cost, and calibration error. We then propose a general framework for active Bayesian assessment using inferred uncertainty to guide efficient selection of instances for labeling, enabling better performance assessment with fewer labels. We demonstrate significant gains from our proposed active Bayesian approach via a series of systematic empirical experiments assessing the performance of modern neural classifiers (e.g., ResNet and BERT) on several standard image and text classification datasets.