# MalBERT: Using Transformers for Cybersecurity and Malicious Software Detection

arXiv: https://arxiv.org/abs/2103.03806 · HTML: https://mlsanity.com/paper/2103.03806/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Java Decompiler Diversity and its Application to Meta-decompilation](https://mlsanity.com/paper/2005.11315/) (2020)

## Built on by (most cited first, 12 of 12)

- [SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security](https://mlsanity.com/paper/2312.15838/) (2023)
- [CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain](https://mlsanity.com/paper/2212.02974/) (2022)
- [On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions](https://mlsanity.com/paper/2306.14062/) (2023)
- [DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification](https://mlsanity.com/paper/2409.09143/) (2024)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)
- [Accelerating Malware Classification: A Vision Transformer Solution](https://mlsanity.com/paper/2409.19461/) (2024)
- [Detection of LLM-Generated Java Code Using Discretized Nested Bigrams](https://mlsanity.com/paper/2502.15740/) (2025)
- [Empirical Evaluation of Concept Drift in ML-Based Android Malware Detection](https://mlsanity.com/paper/2507.22772/) (2025)
- [SecureReg: Combining NLP and MLP for Enhanced Detection of Malicious Domain Name Registrations](https://mlsanity.com/paper/2401.03196/) (2024)
- [Trident: Improving Malware Detection with LLMs and Behavioral Features](https://mlsanity.com/paper/2605.00297/) (2026)
- [The Efficacy of Transformer-based Adversarial Attacks in Security Domains](https://mlsanity.com/paper/2310.11597/) (2023)
- [Advancing NLP Security by Leveraging LLMs as Adversarial Engines](https://mlsanity.com/paper/2410.18215/) (2024)

## Abstract (arXiv)

In recent years we have witnessed an increase in cyber threats and malicious software attacks on different platforms with important consequences to persons and businesses. It has become critical to find automated machine learning techniques to proactively defend against malware. Transformers, a category of attention-based deep learning techniques, have recently shown impressive results in solving different tasks mainly related to the field of Natural Language Processing (NLP). In this paper, we propose the use of a Transformers' architecture to automatically detect malicious software. We propose a model based on BERT (Bidirectional Encoder Representations from Transformers) which performs a static analysis on the source code of Android applications using preprocessed features to characterize existing malware and classify it into different representative malware categories. The obtained results are promising and show the high performance obtained by Transformer-based models for malicious software detection.