# A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels

arXiv: https://arxiv.org/abs/2109.11126 · HTML: https://mlsanity.com/paper/2109.11126/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 58 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [Do We Train on Test Data? Purging CIFAR of Near-Duplicates](https://mlsanity.com/paper/1902.00423/) (2019)
- [Microsoft Malware Classification Challenge](https://mlsanity.com/paper/1802.10135/) (2018)
- [A Survey of Machine Learning Methods and Challenges for Windows Malware Classification](https://mlsanity.com/paper/2006.09271/) (2020)
- [Identifying Statistical Bias in Dataset Replication](https://mlsanity.com/paper/2005.09619/) (2020)
- [Leveraging Uncertainty for Improved Static Malware Detection Under Extreme False Positive Constraints](https://mlsanity.com/paper/2108.04081/) (2021)
- [Unveiling Zeus](https://mlsanity.com/paper/1303.7012/) (2013)

## Built on by (most cited first, 16 of 16)

- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels](https://mlsanity.com/paper/2111.15031/) (2021)
- [SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2305.00550/) (2023)
- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [The Cross-evaluation of Machine Learning-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2203.04686/) (2022)
- [Efficient Malware Analysis Using Metric Embeddings](https://mlsanity.com/paper/2212.02663/) (2022)
- [Is Function Similarity Over-Engineered? Building a Benchmark](https://mlsanity.com/paper/2410.22677/) (2024)
- [Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective](https://mlsanity.com/paper/2302.08287/) (2023)
- [Logical Assessment Formula and Its Principles for Evaluations with Inaccurate Ground-Truth Labels](https://mlsanity.com/paper/2110.11567/) (2021)
- [Living off the Analyst: Harvesting Features from Yara Rules for Malware Detection](https://mlsanity.com/paper/2411.18516/) (2024)
- [Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora](https://mlsanity.com/paper/2511.13808/) (2025)
- [Cybersecurity is the True Frontier for Generative AI Success or Failure](https://mlsanity.com/paper/2606.28929/) (2026)
- [Firenze: Model Evaluation Using Weak Signals](https://mlsanity.com/paper/2207.00827/) (2022)
- [Validation of the Practicability of Logical Assessment Formula for Evaluations with Inaccurate Ground-Truth Labels: An Application Study on Tumour Segmentation for Breast Cancer](https://mlsanity.com/paper/2307.02709/) (2023)
- [High-resolution Image-based Malware Classification using Multiple Instance Learning](https://mlsanity.com/paper/2311.12760/) (2023)
- [Hamm-Grams: An Algorithm for Mining Regular Expressions of Bytes](https://mlsanity.com/paper/2607.01445/) (2026)

## Abstract (arXiv)

In some problem spaces, the high cost of obtaining ground truth labels necessitates use of lower quality reference datasets. It is difficult to benchmark model performance using these datasets, as evaluation results may be biased. We propose a supplement to using reference labels, which we call an approximate ground truth refinement (AGTR). Using an AGTR, we prove that bounds on specific metrics used to evaluate clustering algorithms and multi-class classifiers can be computed without reference labels. We also introduce a procedure that uses an AGTR to identify inaccurate evaluation results produced from datasets of dubious quality. Creating an AGTR requires domain knowledge, and malware family classification is a task with robust domain knowledge approaches that support the construction of an AGTR. We demonstrate our AGTR evaluation framework by applying it to a popular malware labeling tool to diagnose over-fitting in prior testing and evaluate changes whose impact could not be meaningfully quantified under previous data.