# Towards a Fair Comparison and Realistic Evaluation Framework of Android Malware Detectors based on Static Analysis and Machine Learning

arXiv: https://arxiv.org/abs/2205.12569 · HTML: https://mlsanity.com/paper/2205.12569/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 78 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 17 of 17)

- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Adversarial Examples: Opportunities and Challenges](https://mlsanity.com/paper/1809.04790/) (2018)
- [Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detection](https://mlsanity.com/paper/1704.08996/) (2017)
- [MaMaDroid: Detecting Android Malware by Building Markov Chains of Behavioral Models](https://mlsanity.com/paper/1612.04433/) (2016)
- [Reviewer Integration and Performance Measurement for Malware Detection](https://mlsanity.com/paper/1510.07338/) (2015)
- [AVClass2: Massive Malware Tag Extraction from AV Labels](https://mlsanity.com/paper/2006.10615/) (2020)
- [Context-aware, Adaptive and Scalable Android Malware Detection through Online Learning (extended version)](https://mlsanity.com/paper/1706.00947/) (2017)
- [Survey of Network Intrusion Detection Methods from the Perspective of the Knowledge Discovery in Databases Process](https://mlsanity.com/paper/2001.09697/) (2020)
- [Malware Dynamic Analysis Evasion Techniques: A Survey](https://mlsanity.com/paper/1811.01190/) (2018)
- [Android Malware Clustering through Malicious Payload Mining](https://mlsanity.com/paper/1707.04795/) (2017)
- [Eight Years of Rider Measurement in the Android Malware Ecosystem: Evolution and Lessons Learned](https://mlsanity.com/paper/1801.08115/) (2018)
- [Don't Pick the Cherry: An Evaluation Methodology for Android Malware Detection Methods](https://mlsanity.com/paper/1903.10560/) (2019)
- [A two-steps approach to improve the performance of Android malware detectors](https://mlsanity.com/paper/2205.08265/) (2022)
- [Maat: Automatically Analyzing VirusTotal for Accurate Labeling and Effective Malware Detection](https://mlsanity.com/paper/2007.00510/) (2020)
- [On Impact of Semantically Similar Apps in Android Malware Datasets](https://mlsanity.com/paper/2112.02606/) (2021)

## Built on by (most cited first, 6 of 6)

- [Efficient Concept Drift Handling for Batch Android Malware Detection Models](https://mlsanity.com/paper/2309.09807/) (2023)
- [Assessing the Impact of Packing on Machine Learning-Based Malware Detection and Classification Systems](https://mlsanity.com/paper/2410.24017/) (2024)
- [Light up that Droid! On the Effectiveness of Static Analysis Features against App Obfuscation for Android Malware Detection](https://mlsanity.com/paper/2310.15645/) (2023)
- [Comparative Analysis of Hash-based Malware Clustering via K-Means](https://mlsanity.com/paper/2512.09539/) (2025)
- [DeepTrust: Multi-Step Classification through Dissimilar Adversarial Representations for Robust Android Malware Detection](https://mlsanity.com/paper/2510.12310/) (2025)
- [LCC-LLM: Leveraging Code-Centric Large Language Models for Malware Attribution](https://mlsanity.com/paper/2605.05807/) (2026)

## Abstract (arXiv)

As in other cybersecurity areas, machine learning (ML) techniques have emerged as a promising solution to detect Android malware. In this sense, many proposals employing a variety of algorithms and feature sets have been presented to date, often reporting impresive detection performances. However, the lack of reproducibility and the absence of a standard evaluation framework make these proposals difficult to compare. In this paper, we perform an analysis of 10 influential research works on Android malware detection using a common evaluation framework. We have identified five factors that, if not taken into account when creating datasets and designing detectors, significantly affect the trained ML models and their performances. In particular, we analyze the effect of (1) the presence of duplicated samples, (2) label (goodware/greyware/malware) attribution, (3) class imbalance, (4) the presence of apps that use evasion techniques and, (5) the evolution of apps. Based on this extensive experimentation, we conclude that the studied ML-based detectors have been evaluated optimistically, which justifies the good published results. Our findings also highlight that it is imperative to generate realistic experimental scenarios, taking into account the aforementioned factors, to foster the rise of better ML-based Android malware detection solutions.