# Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach

arXiv: https://arxiv.org/abs/1705.07086 · HTML: https://mlsanity.com/paper/1705.07086/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2017.

## Built on by (most cited first, 27 of 27)

- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](https://mlsanity.com/paper/2106.05961/) (2021)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [Self-Training with Weak Supervision](https://mlsanity.com/paper/2104.05514/) (2021)
- [Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation](https://mlsanity.com/paper/2302.01094/) (2023)
- [Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models](https://mlsanity.com/paper/2205.06130/) (2022)
- [DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples](https://mlsanity.com/paper/2110.13740/) (2021)
- [Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions](https://mlsanity.com/paper/2402.01108/) (2024)
- [(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy](https://mlsanity.com/paper/2306.00312/) (2023)
- [On the Importance of Feature Separability in Predicting Out-Of-Distribution Error](https://mlsanity.com/paper/2303.15488/) (2023)
- [Predicting Out-of-Distribution Error with Confidence Optimal Transport](https://mlsanity.com/paper/2302.05018/) (2023)
- [LOVM: Language-Only Vision Model Selection](https://mlsanity.com/paper/2306.08893/) (2023)
- [Label-Free Model Evaluation with Semi-Structured Dataset Representations](https://mlsanity.com/paper/2112.00694/) (2021)
- [Ranking Models in Unlabeled New Environments](https://mlsanity.com/paper/2108.10310/) (2021)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [Bounding Box Stability against Feature Dropout Reflects Detector Generalization across Environments](https://mlsanity.com/paper/2403.13803/) (2024)
- [DAPPER: Label-Free Performance Estimation after Personalization for Heterogeneous Mobile Sensing](https://mlsanity.com/paper/2111.11053/) (2021)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings](https://mlsanity.com/paper/2505.22356/) (2025)
- [Beyond Consensus: Trace-Level Synthesis in Mixture of Agents](https://mlsanity.com/paper/2605.29116/) (2026)
- [Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shift](https://mlsanity.com/paper/2401.08909/) (2024)
- [Metric Ensembles For Hallucination Detection](https://mlsanity.com/paper/2310.10495/) (2023)
- [Segmentation of turbulent computational fluid dynamics simulations with unsupervised ensemble learning](https://mlsanity.com/paper/2109.01381/) (2021)
- [Label Augmentation with Reinforced Labeling for Weak Supervision](https://mlsanity.com/paper/2204.06436/) (2022)
- [MetaErr: Towards Predicting Error Patterns in Deep Neural Networks](https://mlsanity.com/paper/2604.23289/) (2026)

## Abstract (arXiv)

We propose an efficient method to estimate the accuracy of classifiers using only unlabeled data. We consider a setting with multiple classification problems where the target classes may be tied together through logical constraints. For example, a set of classes may be mutually exclusive, meaning that a data instance can belong to at most one of them. The proposed method is based on the intuition that: (i) when classifiers agree, they are more likely to be correct, and (ii) when the classifiers make a prediction that violates the constraints, at least one classifier must be making an error. Experiments on four real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs. The results emphasize the utility of logical constraints in estimating accuracy, thus validating our intuition.