mlsanityData to 14 Oct 2026

Towards a Fair Comparison and Realistic Evaluation Framework of Android Malware Detectors based on Static Analysis and Machine Learning

arXiv 2205.12569 ↗First posted 25 May 2022cs.CR cs.LG cs.SEcited by 6

Not on the map. Why?

Abstract (arXiv)

As in other cybersecurity areas, machine learning (ML) techniques have emerged as a promising solution to detect Android malware. In this sense, many proposals employing a variety of algorithms and feature sets have been presented to date, often reporting impresive detection performances. However, the lack of reproducibility and the absence of a standard evaluation framework make these proposals difficult to compare. In this paper, we perform an analysis of 10 influential research works on Android malware detection using a common evaluation framework. We have identified five factors that, if not taken into account when creating datasets and designing detectors, significantly affect the trained ML models and their performances. In particular, we analyze the effect of (1) the presence of duplicated samples, (2) label (goodware/greyware/malware) attribution, (3) class imbalance, (4) the presence of apps that use evasion techniques and, (5) the evolution of apps. Based on this extensive experimentation, we conclude that the studied ML-based detectors have been evaluated optimistically, which justifies the good published results. Our findings also highlight that it is imperative to generate realistic experimental scenarios, taking into account the aforementioned factors, to foster the rise of better ML-based Android malware detection solutions.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 17 of its 78 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

17 of 78 references.

References · 17

  1. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  2. 2020Learning under Concept Drift: A Reviewcited by 477
  3. 2018TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Timecited by 108
  4. 2018Adversarial Examples: Opportunities and Challengescited by 63
  5. 2017Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detectioncited by 52
  6. 2016MaMaDroid: Detecting Android Malware by Building Markov Chains of Behavioral Modelscited by 30
  7. 2015Reviewer Integration and Performance Measurement for Malware Detectioncited by 19
  8. 2020AVClass2: Massive Malware Tag Extraction from AV Labelscited by 19
  9. 2017Context-aware, Adaptive and Scalable Android Malware Detection through Online Learning (extended version)cited by 11
  10. 2020Survey of Network Intrusion Detection Methods from the Perspective of the Knowledge Discovery in Databases Processcited by 7
  11. 2018Malware Dynamic Analysis Evasion Techniques: A Surveycited by 6
  12. 2017Android Malware Clustering through Malicious Payload Miningcited by 3
5 more
  1. 2018Eight Years of Rider Measurement in the Android Malware Ecosystem: Evolution and Lessons Learnedcited by 3
  2. 2019Don't Pick the Cherry: An Evaluation Methodology for Android Malware Detection Methodscited by 2
  3. 2022A two-steps approach to improve the performance of Android malware detectorscited by 2
  4. 2020Maat: Automatically Analyzing VirusTotal for Accurate Labeling and Effective Malware Detectioncited by 1
  5. 2021On Impact of Semantically Similar Apps in Android Malware Datasetscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 2; 2026: 1.

6 papers.

  1. 2023Efficient Concept Drift Handling for Batch Android Malware Detection Modelscited by 5
  2. 2024Assessing the Impact of Packing on Machine Learning-Based Malware Detection and Classification Systemscited by 5
  3. 2023Light up that Droid! On the Effectiveness of Static Analysis Features against App Obfuscation for Android Malware Detectioncited by 1
  4. 2025Comparative Analysis of Hash-based Malware Clustering via K-Meanscited by 1
  5. 2025DeepTrust: Multi-Step Classification through Dissimilar Adversarial Representations for Robust Android Malware Detectioncited by 0
  6. 2026LCC-LLM: Leveraging Code-Centric Large Language Models for Malware Attributioncited by 0