# Man versus Machine: AutoML and Human Experts' Role in Phishing Detection

arXiv: https://arxiv.org/abs/2108.12193 · HTML: https://mlsanity.com/paper/2108.12193/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks](https://mlsanity.com/paper/1810.10863/) (2018)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science](https://mlsanity.com/paper/1603.06212/) (2016)
- [Understanding data augmentation for classification: when to warp?](https://mlsanity.com/paper/1609.08764/) (2016)
- [How Complex is your classification problem? A survey on measuring classification complexity](https://mlsanity.com/paper/1808.03591/) (2018)
- [Learning To Simulate](https://mlsanity.com/paper/1810.02513/) (2018)
- [Learning-based Video Motion Magnification](https://mlsanity.com/paper/1804.02684/) (2018)
- [Online Hyper-parameter Learning for Auto-Augmentation Strategy](https://mlsanity.com/paper/1905.07373/) (2019)
- [UniformAugment: A Search-free Probabilistic Data Augmentation Approach](https://mlsanity.com/paper/2003.14348/) (2020)
- [Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools](https://mlsanity.com/paper/1908.05557/) (2019)
- [BoostClean: Automated Error Detection and Repair for Machine Learning](https://mlsanity.com/paper/1711.01299/) (2017)
- [Relational Autoencoder for Feature Extraction](https://mlsanity.com/paper/1802.03145/) (2018)
- [AlphaClean: Automatic Generation of Data Cleaning Pipelines](https://mlsanity.com/paper/1904.11827/) (2019)
- [Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop](https://mlsanity.com/paper/2101.04296/) (2021)
- [SOK: A Comprehensive Reexamination of Phishing Research from the Security Perspective](https://mlsanity.com/paper/1911.00953/) (2019)
- [Style transfer-based image synthesis as an efficient regularization technique in deep learning](https://mlsanity.com/paper/1905.10974/) (2019)

## Abstract (arXiv)

Machine learning (ML) has developed rapidly in the past few years and has successfully been utilized for a broad range of tasks, including phishing detection. However, building an effective ML-based detection system is not a trivial task, and requires data scientists with knowledge of the relevant domain. Automated Machine Learning (AutoML) frameworks have received a lot of attention in recent years, enabling non-ML experts in building a machine learning model. This brings to an intriguing question of whether AutoML can outperform the results achieved by human data scientists. Our paper compares the performances of six well-known, state-of-the-art AutoML frameworks on ten different phishing datasets to see whether AutoML-based models can outperform manually crafted machine learning models. Our results indicate that AutoML-based models are able to outperform manually developed machine learning models in complex classification tasks, specifically in datasets where the features are not quite discriminative, and datasets with overlapping classes or relatively high degrees of non-linearity. Challenges also remain in building a real-world phishing detection system using AutoML frameworks due to the current support only on supervised classification problems, leading to the need for labeled data, and the inability to update the AutoML-based models incrementally. This indicates that experts with knowledge in the domain of phishing and cybersecurity are still essential in the loop of the phishing detection pipeline.