# A Lean Transformer Model for Dynamic Malware Analysis and Detection

arXiv: https://arxiv.org/abs/2408.02313 · HTML: https://mlsanity.com/paper/2408.02313/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [Spear Phishing With Large Language Models](https://mlsanity.com/paper/2305.06972/) (2023)
- [Dynamic Malware Analysis with Feature Engineering and Feature Learning](https://mlsanity.com/paper/1907.07352/) (2019)
- [Neurlux: Dynamic Malware Analysis Without Feature Engineering](https://mlsanity.com/paper/1910.11376/) (2019)
- [On the Effectiveness of Pretrained Models for API Learning](https://mlsanity.com/paper/2204.03498/) (2022)
- [Quo Vadis: Hybrid Machine Learning Meta-Model based on Contextual and Behavioral Malware Representations](https://mlsanity.com/paper/2208.12248/) (2022)
- [MalFox: Camouflaged Adversarial Malware Example Generation Based on Conv-GANs Against Black-Box Detectors](https://mlsanity.com/paper/2011.01509/) (2020)
- [AI-based Malware and Ransomware Detection Models](https://mlsanity.com/paper/2207.02108/) (2022)

## Abstract (arXiv)

Malware is a fast-growing threat to the modern computing world and existing lines of defense are not efficient enough to address this issue. This is mainly due to the fact that many prevention solutions rely on signature-based detection methods that can easily be circumvented by hackers. Therefore, there is a recurrent need for behavior-based analysis where a suspicious file is ran in a secured environment and its traces are collected to reports for analysis. Previous works have shown some success leveraging Neural Networks and API calls sequences extracted from these execution reports. Recently, Large Language Models and Generative AI have demonstrated impressive capabilities mainly in Natural Language Processing tasks and promising applications in the cybersecurity field for both attackers and defenders. In this paper, we design an Encoder-Only model, based on the Transformers architecture, to detect malicious files, digesting their API call sequences collected by an execution emulation solution. We are also limiting the size of the model architecture and the number of its parameters since it is often considered that Large Language Models may be overkill for specific tasks such as the one we are dealing with hereafter. In addition to achieving decent detection results, this approach has the advantage of reducing our carbon footprint by limiting training and inference times and facilitating technical operations with less hardware requirements. We also carry out some analysis of our results and highlight the limits and possible improvements when using Transformers to analyze malicious files.