# Don't Pick the Cherry: An Evaluation Methodology for Android Malware Detection Methods

arXiv: https://arxiv.org/abs/1903.10560 · HTML: https://mlsanity.com/paper/1903.10560/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2019.

## Built on by (most cited first, 2 of 2)

- [Towards a Fair Comparison and Realistic Evaluation Framework of Android Malware Detectors based on Static Analysis and Machine Learning](https://mlsanity.com/paper/2205.12569/) (2022)
- [Efficient Concept Drift Handling for Batch Android Malware Detection Models](https://mlsanity.com/paper/2309.09807/) (2023)

## Abstract (arXiv)

In evaluating detection methods, the malware research community relies on scan results obtained from online platforms such as VirusTotal. Nevertheless, given the lack of standards on how to interpret the obtained data to label apps, researchers hinge on their intuitions and adopt different labeling schemes. The dynamicity of VirusTotal's results along with adoption of different labeling schemes significantly affect the accuracies achieved by any given detection method even on the same dataset, which gives subjective views on the method's performance and hinders the comparison of different malware detection techniques. In this paper, we demonstrate the effect of varying (1) time, (2) labeling schemes, and (3) attack scenarios on the performance of an ensemble of Android repackaged malware detection methods, called dejavu, using over 30,000 real-world Android apps. Our results vividly show the impact of varying the aforementioned 3 dimensions on dejavu's performance. With such results, we encourage the adoption of a standard methodology that takes into account those 3 dimensions in evaluating newly-devised methods to detect Android (repackaged) malware.