# Estimating the Accuracies of Multiple Classifiers Without Labeled Data

arXiv: https://arxiv.org/abs/1407.7644 · HTML: https://mlsanity.com/paper/1407.7644/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2014.

## Built on by (most cited first, 17 of 17)

- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples](https://mlsanity.com/paper/2110.13740/) (2021)
- [Anomaly Detection with Variance Stabilized Density Estimation](https://mlsanity.com/paper/2306.00582/) (2023)
- [Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings](https://mlsanity.com/paper/2505.22356/) (2025)
- [FUSE: Ensembling Verifiers with Zero Labeled Data](https://mlsanity.com/paper/2604.18547/) (2026)
- [Learning From Crowdsourced Noisy Labels: A Signal Processing Perspective](https://mlsanity.com/paper/2407.06902/) (2024)
- [Memory Reward Inflation in Self-Improving LLM Agents](https://mlsanity.com/paper/2608.00017/) (2026)
- [Seed Word Selection for Weakly-Supervised Text Classification with Unsupervised Error Estimation](https://mlsanity.com/paper/2104.09765/) (2021)
- [Streaming algorithms for evaluating noisy judges on unlabeled data -- binary classification](https://mlsanity.com/paper/2306.01726/) (2023)
- [Sharp bounds on aggregate expert error](https://mlsanity.com/paper/2407.16642/) (2024)
- [Detecting adversaries in Crowdsourcing](https://mlsanity.com/paper/2110.04117/) (2021)
- [Ranking and Combining Latent Structured Predictive Scores without Labeled Data](https://mlsanity.com/paper/2408.07796/) (2024)
- [A logical alarm for misaligned binary classifiers](https://mlsanity.com/paper/2409.11052/) (2024)
- [Logical Consistency Between Disagreeing Experts and Its Role in AI Safety](https://mlsanity.com/paper/2510.00821/) (2025)
- [SDSR: A Spectral Divide-and-Conquer Approach for Species Tree Reconstruction](https://mlsanity.com/paper/2603.10215/) (2026)

## Abstract (arXiv)

In various situations one is given only the predictions of multiple classifiers over a large unlabeled test data. This scenario raises the following questions: Without any labeled data and without any a-priori knowledge about the reliability of these different classifiers, is it possible to consistently and computationally efficiently estimate their accuracies? Furthermore, also in a completely unsupervised manner, can one construct a more accurate unsupervised ensemble classifier? In this paper, focusing on the binary case, we present simple, computationally efficient algorithms to solve these questions. Furthermore, under standard classifier independence assumptions, we prove our methods are consistent and study their asymptotic error. Our approach is spectral, based on the fact that the off-diagonal entries of the classifiers' covariance matrix and 3-d tensor are rank-one. We illustrate the competitive performance of our algorithms via extensive experiments on both artificial and real datasets.