# Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora

arXiv: https://arxiv.org/abs/2511.13808 · HTML: https://mlsanity.com/paper/2511.13808/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 71 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time](https://mlsanity.com/paper/1807.07838/) (2018)
- [Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features](https://mlsanity.com/paper/1508.03096/) (2015)
- [A Survey of Machine Learning Methods and Challenges for Windows Malware Classification](https://mlsanity.com/paper/2006.09271/) (2020)
- [Survey of Machine Learning Techniques for Malware Analysis](https://mlsanity.com/paper/1710.08189/) (2017)
- [EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers](https://mlsanity.com/paper/2506.05074/) (2025)
- [Automatic Yara Rule Generation Using Biclustering](https://mlsanity.com/paper/2009.03779/) (2020)
- [MOTIF: A Large Malware Reference Dataset with Ground Truth Family Labels](https://mlsanity.com/paper/2111.15031/) (2021)
- [A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels](https://mlsanity.com/paper/2109.11126/) (2021)
- [Zipf's law holds for phrases, not words](https://mlsanity.com/paper/1406.5181/) (2014)
- [Assemblage: Automatic Binary Dataset Construction for Machine Learning](https://mlsanity.com/paper/2405.03991/) (2024)
- [ClarAVy: A Tool for Scalable and Accurate Malware Family Labeling](https://mlsanity.com/paper/2502.02759/) (2025)
- [KiloGrams: Very Large N-Grams for Malware Classification](https://mlsanity.com/paper/1908.00200/) (2019)
- [Semi-supervised Classification of Malware Families Under Extreme Class Imbalance via Hierarchical Non-Negative Matrix Factorization with Automatic Model Selection](https://mlsanity.com/paper/2309.06643/) (2023)
- [Rank-1 Similarity Matrix Decomposition For Modeling Changes in Antivirus Consensus Through Time](https://mlsanity.com/paper/2201.00757/) (2021)
- [Efficient Malware Analysis Using Metric Embeddings](https://mlsanity.com/paper/2212.02663/) (2022)
- [Transformers for End-to-End InfoSec Tasks: A Feasibility Study](https://mlsanity.com/paper/2212.02666/) (2022)
- [Is Function Similarity Over-Engineered? Building a Benchmark](https://mlsanity.com/paper/2410.22677/) (2024)
- [On Normalized Compression Distance and Large Malware](https://mlsanity.com/paper/1509.00689/) (2015)
- [AVScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale Malware Corpora](https://mlsanity.com/paper/2306.06228/) (2023)
- [Would a File by Any Other Name Seem as Malicious?](https://mlsanity.com/paper/1910.04753/) (2019)
- [High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates](https://mlsanity.com/paper/2407.06346/) (2024)
- [Living off the Analyst: Harvesting Features from Yara Rules for Malware Detection](https://mlsanity.com/paper/2411.18516/) (2024)
- [A Parallel Space Saving Algorithm For Frequent Items and the Hurwitz zeta distribution](https://mlsanity.com/paper/1401.0702/) (2014)
- [Smooth q-Gram, and Its Applications to Detection of Overlaps among Long, Error-Prone Sequencing Reads](https://mlsanity.com/paper/1802.01189/) (2018)

## Built on by (most cited first, 2 of 2)

- [Hamm-Grams: An Algorithm for Mining Regular Expressions of Bytes](https://mlsanity.com/paper/2607.01445/) (2026)
- [Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets](https://mlsanity.com/paper/2609.23995/) (2026)

## Abstract (arXiv)

A classifier using byte n-grams as features is the only approach we have found fast enough to meet requirements in size (sub 2 MB), speed (multiple GB/s), and latency (sub 10 ms) for deployment in numerous malware detection scenarios. However, we've consistently found that 6-8 grams achieve the best accuracy on our production deployments but have been unable to deploy regularly updated models due to the high cost of finding the top-k most frequent n-grams over terabytes of executable programs. Because the Zipfian distribution well models the distribution of n-grams, we exploit its properties to develop a new top-k n-gram extractor that is up to 35× faster than the previous best alternative. Using our new Zipf-Gramming algorithm, we are able to scale up our production training set and obtain up to 30% improvement in AUC at detecting new malware. We show theoretically and empirically that our approach will select the top-k items with little error and the interplay between theory and engineering required to achieve these results.