# EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers

arXiv: https://arxiv.org/abs/2506.05074 · HTML: https://mlsanity.com/paper/2506.05074/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 32 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 7 of 7)

- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection](https://mlsanity.com/paper/2012.07634/) (2020)
- [Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features](https://mlsanity.com/paper/1508.03096/) (2015)
- [A Survey of Machine Learning Methods and Challenges for Windows Malware Classification](https://mlsanity.com/paper/2006.09271/) (2020)
- [ClarAVy: A Tool for Scalable and Accurate Malware Family Labeling](https://mlsanity.com/paper/2502.02759/) (2025)
- [Towards an Automated Pipeline for Detecting and Classifying Malware through Machine Learning](https://mlsanity.com/paper/2106.05625/) (2021)
- [MalDICT: Benchmark Datasets on Malware Behaviors, Platforms, Exploitation, and Packers](https://mlsanity.com/paper/2310.11706/) (2023)

## Built on by (most cited first, 19 of 19)

- [Semantic Preprocessing for LLM-based Malware Analysis](https://mlsanity.com/paper/2506.12113/) (2025)
- [Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora](https://mlsanity.com/paper/2511.13808/) (2025)
- [Cybersecurity is the True Frontier for Generative AI Success or Failure](https://mlsanity.com/paper/2606.28929/) (2026)
- [Adversarial Defense in Cybersecurity: A Systematic Review of GANs for Threat Detection and Mitigation](https://mlsanity.com/paper/2509.20411/) (2025)
- [Binary-30K: A Heterogeneous Dataset for Deep Learning in Binary Analysis and Malware Detection](https://mlsanity.com/paper/2511.22095/) (2025)
- [Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models](https://mlsanity.com/paper/2604.02490/) (2026)
- [Trident: Improving Malware Detection with LLMs and Behavioral Features](https://mlsanity.com/paper/2605.00297/) (2026)
- [Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach](https://mlsanity.com/paper/2607.10534/) (2026)
- [CIC-Trap4Phish: A Unified Multi-Format Dataset for Phishing and Quishing Attachment Detection](https://mlsanity.com/paper/2602.09015/) (2026)
- [Detecting Concept Drift in Evolving Malware Families Using Rule-Based Classifier Representations](https://mlsanity.com/paper/2604.22629/) (2026)
- [Gray-Box Poisoning of Continuous Malware Ingestion Pipelines](https://mlsanity.com/paper/2605.04698/) (2026)
- [McNdroid: A Longitudinal Multimodal Benchmark for Robust Drift Detection in Android Malware](https://mlsanity.com/paper/2605.06894/) (2026)
- [ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage](https://mlsanity.com/paper/2605.21615/) (2026)
- [Building an Adversarial Malware Dataset by Family and Type: Generation, Evasion, and Poisoning Evaluation](https://mlsanity.com/paper/2605.25937/) (2026)
- [A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution](https://mlsanity.com/paper/2606.30572/) (2026)
- [EXE-Bench: Ranking the Tradeoffs of AI-based Windows Malware Detectors for Real-World Usability](https://mlsanity.com/paper/2607.24177/) (2026)
- [Latent Stability Analysis of Malware Representations Under Feature-Space Perturbations](https://mlsanity.com/paper/2607.24896/) (2026)
- [When LLM Decompilers Recompile More and Preserve Less](https://mlsanity.com/paper/2609.05370/) (2026)
- [Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets](https://mlsanity.com/paper/2609.23995/) (2026)

## Abstract (arXiv)

A lack of accessible data has historically restricted malware analysis research, and practitioners have relied heavily on datasets provided by industry sources to advance. Existing public datasets are limited by narrow scope - most include files targeting a single platform, have labels supporting just one type of malware classification task, and make no effort to capture the evasive files that make malware detection difficult in practice. We present EMBER2024, a new dataset that enables holistic evaluation of malware classifiers. Created in collaboration with the authors of EMBER2017 and EMBER2018, the EMBER2024 dataset includes hashes, metadata, feature vectors, and labels for more than 3.2 million files from six file formats. Our dataset supports the training and evaluation of machine learning models on seven malware classification tasks, including malware detection, malware family classification, and malware behavior identification. EMBER2024 is the first to include a collection of malicious files that initially went undetected by a set of antivirus products, creating a "challenge" set to assess classifier performance against evasive malware. This work also introduces EMBER feature version 3, with added support for several new feature types. We are releasing the EMBER2024 dataset to promote reproducibility and empower researchers in the pursuit of new malware research topics.