# An Ensemble of Pre-trained Transformer Models For Imbalanced Multiclass Malware Classification

arXiv: https://arxiv.org/abs/2112.13236 · HTML: https://mlsanity.com/paper/2112.13236/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 25 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [Compressing Large-Scale Transformer-Based Models: A Case Study on BERT](https://mlsanity.com/paper/2002.11985/) (2020)
- [CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters](https://mlsanity.com/paper/2010.10392/) (2020)
- [Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification](https://mlsanity.com/paper/2012.03173/) (2020)
- [Survey of Machine Learning Techniques for Malware Analysis](https://mlsanity.com/paper/1710.08189/) (2017)
- [Neurlux: Dynamic Malware Analysis Without Feature Engineering](https://mlsanity.com/paper/1910.11376/) (2019)
- [Random CapsNet Forest Model for Imbalanced Malware Type Classification Task](https://mlsanity.com/paper/1912.10836/) (2019)
- [A Benchmark API Call Dataset for Windows PE Malware Classification](https://mlsanity.com/paper/1905.01999/) (2019)

## Built on by (most cited first, 5 of 5)

- [A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation](https://mlsanity.com/paper/2304.02858/) (2023)
- [Nebula: Self-Attention for Dynamic Malware Analysis](https://mlsanity.com/paper/2310.10664/) (2023)
- [Prompt Engineering-assisted Malware Dynamic Analysis Using GPT-4](https://mlsanity.com/paper/2312.08317/) (2023)
- [Unlocking Real-Time Fluorescence Lifetime Imaging: Multi-Pixel Parallelism for FPGA-Accelerated Processing](https://mlsanity.com/paper/2410.07364/) (2024)
- [Ensemble BERT for Medication Event Classification on Electronic Health Records (EHRs)](https://mlsanity.com/paper/2506.23315/) (2025)

## Abstract (arXiv)

Classification of malware families is crucial for a comprehensive understanding of how they can infect devices, computers, or systems. Thus, malware identification enables security researchers and incident responders to take precautions against malware and accelerate mitigation. API call sequences made by malware are widely utilized features by machine and deep learning models for malware classification as these sequences represent the behavior of malware. However, traditional machine and deep learning models remain incapable of capturing sequence relationships between API calls. On the other hand, the transformer-based models process sequences as a whole and learn relationships between API calls due to multi-head attention mechanisms and positional embeddings. Our experiments demonstrate that the transformer model with one transformer block layer surpassed the widely used base architecture, LSTM. Moreover, BERT or CANINE, pre-trained transformer models, outperformed in classifying highly imbalanced malware families according to evaluation metrics, F1-score, and AUC score. Furthermore, the proposed bagging-based random transformer forest (RTF), an ensemble of BERT or CANINE, has reached the state-of-the-art evaluation scores on three out of four datasets, particularly state-of-the-art F1-score of 0.6149 on one of the commonly used benchmark dataset.