# Ranking and combining multiple predictors without labeled data

arXiv: https://arxiv.org/abs/1303.3257 · HTML: https://mlsanity.com/paper/1303.3257/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2013.

## Built on by (most cited first, 22 of 22)

- [Crowd-PrefRL: Preference-Based Reward Learning from Crowds](https://mlsanity.com/paper/2401.10941/) (2024)
- [A Spectral Method for Assessing and Combining Multiple Data Visualizations](https://mlsanity.com/paper/2210.13711/) (2022)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [FUSE: Ensembling Verifiers with Zero Labeled Data](https://mlsanity.com/paper/2604.18547/) (2026)
- [Streaming algorithms for evaluating noisy judges on unlabeled data -- binary classification](https://mlsanity.com/paper/2306.01726/) (2023)
- [Scalable Interactive Machine Learning for Future Command and Control](https://mlsanity.com/paper/2402.06501/) (2024)
- [Sharp bounds on aggregate expert error](https://mlsanity.com/paper/2407.16642/) (2024)
- [Quantifying Human Bias and Knowledge to guide ML models during Training](https://mlsanity.com/paper/2211.10796/) (2022)
- [An Adaptive Method for Weak Supervision with Drifting Data](https://mlsanity.com/paper/2306.01658/) (2023)
- [The logic of NTQR evaluations of noisy AI agents: Complete postulates and logically consistent error correlations](https://mlsanity.com/paper/2312.05392/) (2023)
- [Copula Quadrant Similarity for Anomaly Scores](https://mlsanity.com/paper/2101.02330/) (2021)
- [Blind Exploration and Exploitation of Stochastic Experts](https://mlsanity.com/paper/2104.01078/) (2021)
- [Gaussian Graphical Models as an Ensemble Method for Distributed Gaussian Processes](https://mlsanity.com/paper/2202.03287/) (2022)
- [Entry Dependent Expert Selection in Distributed Gaussian Processes Using Multilabel Classification](https://mlsanity.com/paper/2211.09940/) (2022)
- [Unsupervised Opinion Aggregation -- A Statistical Perspective](https://mlsanity.com/paper/2308.10386/) (2023)
- [Ranking and Combining Latent Structured Predictive Scores without Labeled Data](https://mlsanity.com/paper/2408.07796/) (2024)
- [Federated One-Shot Ensemble Clustering](https://mlsanity.com/paper/2409.08396/) (2024)
- [A logical alarm for misaligned binary classifiers](https://mlsanity.com/paper/2409.11052/) (2024)
- [Logical Consistency Between Disagreeing Experts and Its Role in AI Safety](https://mlsanity.com/paper/2510.00821/) (2025)
- [StackingNet: Collective Inference Across Independent AI Foundation Models](https://mlsanity.com/paper/2602.13792/) (2026)
- [The Metanym Game: An LLM Benchmark Without Ground Truth That Rises With the Models It Measures](https://mlsanity.com/paper/2606.21008/) (2026)
- [Fold'EM: Direct atomic structure inference from Cryo-EM particles](https://mlsanity.com/paper/2610.01358/) (2026)

## Abstract (arXiv)

In a broad range of classification and decision making problems, one is given the advice or predictions of several classifiers, of unknown reliability, over multiple questions or queries. This scenario is different from the standard supervised setting, where each classifier accuracy can be assessed using available labeled data, and raises two questions: given only the predictions of several classifiers over a large set of unlabeled test data, is it possible to a) reliably rank them; and b) construct a meta-classifier more accurate than most classifiers in the ensemble? Here we present a novel spectral approach to address these questions. First, assuming conditional independence between classifiers, we show that the off-diagonal entries of their covariance matrix correspond to a rank-one matrix. Moreover, the classifiers can be ranked using the leading eigenvector of this covariance matrix, as its entries are proportional to their balanced accuracies. Second, via a linear approximation to the maximum likelihood estimator, we derive the Spectral Meta-Learner (SML), a novel ensemble classifier whose weights are equal to this eigenvector entries. On both simulated and real data, SML typically achieves a higher accuracy than most classifiers in the ensemble and can provide a better starting point than majority voting, for estimating the maximum likelihood solution. Furthermore, SML is robust to the presence of small malicious groups of classifiers designed to veer the ensemble prediction away from the (unknown) ground truth.