# SoK: The Impact of Unlabelled Data in Cyberthreat Detection

arXiv: https://arxiv.org/abs/2205.08944 · HTML: https://mlsanity.com/paper/2205.08944/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 157 references that are in the map.
- First posted on arXiv 18 May 2022.

## Builds on (most cited first, 26 of 26)

- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection](https://mlsanity.com/paper/1802.09089/) (2018)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Exploring Connections Between Active Learning and Model Extraction](https://mlsanity.com/paper/1811.02054/) (2018)
- [Poisoning the Unlabeled Dataset of Semi-Supervised Learning](https://mlsanity.com/paper/2105.01622/) (2021)
- [XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning](https://mlsanity.com/paper/1912.00290/) (2019)
- [Reviewer Integration and Performance Measurement for Malware Detection](https://mlsanity.com/paper/1510.07338/) (2015)
- [Active Learning for Wireless IoT Intrusion Detection](https://mlsanity.com/paper/1808.01412/) (2018)
- [SOK: A Comprehensive Reexamination of Phishing Research from the Security Perspective](https://mlsanity.com/paper/1911.00953/) (2019)
- [DeltaPhish: Detecting Phishing Webpages in Compromised Websites](https://mlsanity.com/paper/1707.00317/) (2017)
- [Hardening Random Forest Cyber Detectors Against Adversarial Attacks](https://mlsanity.com/paper/1912.03790/) (2019)
- [Random Partitioning Forest for Point-Wise and Collective Anomaly Detection -- Application to Intrusion Detection](https://mlsanity.com/paper/2006.16801/) (2020)
- [Darknet and Deepnet Mining for Proactive Cybersecurity Threat Intelligence](https://mlsanity.com/paper/1607.08583/) (2016)
- [On generating network traffic datasets with synthetic attacks for intrusion detection](https://mlsanity.com/paper/1905.00304/) (2019)
- [Experimental Review of Neural-based approaches for Network Intrusion Management](https://mlsanity.com/paper/2009.09011/) (2020)
- [Optimal Sample Size Planning for the Wilcoxon-Mann-Whitney-Test](https://mlsanity.com/paper/1805.12249/) (2018)
- [Deep Learning for Malicious Flow Detection](https://mlsanity.com/paper/1802.03358/) (2018)
- [How Did That Get In My Phone? Unwanted App Distribution on Android Devices](https://mlsanity.com/paper/2010.10088/) (2020)
- [Semi-supervised classification for dynamic Android malware detection](https://mlsanity.com/paper/1704.05948/) (2017)

## Built on by (most cited first, 15 of 15)

- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [The Role of Machine Learning in Cybersecurity](https://mlsanity.com/paper/2206.09707/) (2022)
- [SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2305.00550/) (2023)
- [A Survey on Malware Detection with Graph Representation Learning](https://mlsanity.com/paper/2303.16004/) (2023)
- [Nebula: Self-Attention for Dynamic Malware Analysis](https://mlsanity.com/paper/2310.10664/) (2023)
- [Multi-SpacePhish: Extending the Evasion-space of Adversarial Attacks against Phishing Website Detectors using Machine Learning](https://mlsanity.com/paper/2210.13660/) (2022)
- [SoK: On the Offensive Potential of AI](https://mlsanity.com/paper/2412.18442/) (2024)
- [Cybersecurity is the True Frontier for Generative AI Success or Failure](https://mlsanity.com/paper/2606.28929/) (2026)
- [AnomalyAID: Reliable Interpretation for Semi-supervised Network Anomaly Detection](https://mlsanity.com/paper/2411.11293/) (2024)
- [ADAPT: A Pseudo-labeling Approach to Combat Concept Drift in Malware Detection](https://mlsanity.com/paper/2507.08597/) (2025)
- [Assessing the Impact of a Supervised Classification Filter on Flow-based Hybrid Network Anomaly Detection](https://mlsanity.com/paper/2310.06656/) (2023)
- [Introducing Adaptive Continuous Adversarial Training (ACAT) to Enhance ML Robustness](https://mlsanity.com/paper/2403.10461/) (2024)
- [MLRan: A Behavioural Dataset for Ransomware Analysis and Detection](https://mlsanity.com/paper/2505.18613/) (2025)
- [An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detection](https://mlsanity.com/paper/2510.02349/) (2025)
- [Multi-Level Distributional Entropy from Flow Summary Statistics for Explainable Network Intrusion Detection](https://mlsanity.com/paper/2606.29797/) (2026)

## Abstract (arXiv)

Machine learning (ML) has become an important paradigm for cyberthreat detection (CTD) in the recent years. A substantial research effort has been invested in the development of specialized algorithms for CTD tasks. From the operational perspective, however, the progress of ML-based CTD is hindered by the difficulty in obtaining the large sets of labelled data to train ML detectors. A potential solution to this problem are semisupervised learning (SsL) methods, which combine small labelled datasets with large amounts of unlabelled data. This paper is aimed at systematization of existing work on SsL for CTD and, in particular, on understanding the utility of unlabelled data in such systems. To this end, we analyze the cost of labelling in various CTD tasks and develop a formal cost model for SsL in this context. Building on this foundation, we formalize a set of requirements for evaluation of SsL methods, which elucidates the contribution of unlabelled data. We review the state-of-the-art and observe that no previous work meets such requirements. To address this problem, we propose a framework for assessing the benefits of unlabelled data in SsL. We showcase an application of this framework by performing the first benchmark evaluation that highlights the tradeoffs of 9 existing SsL methods on 9 public datasets. Our findings verify that, in some cases, unlabelled data provides a small, but statistically significant, performance gain. This paper highlights that SsL in CTD has a lot of room for improvement, which should stimulate future research in this field.