# Transfer Learning in Pre-Trained Large Language Models for Malware Detection Based on System Calls

arXiv: https://arxiv.org/abs/2405.09318 · HTML: https://mlsanity.com/paper/2405.09318/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 15 May 2024.

## Builds on (most cited first, 7 of 7)

- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense](https://mlsanity.com/paper/2303.12132/) (2023)
- [Electrosense: Open and Big Spectrum Data](https://mlsanity.com/paper/1703.09989/) (2017)
- [LSTM-Based System-Call Language Modeling and Robust Ensemble Method for Designing Host-Based Intrusion Detection Systems](https://mlsanity.com/paper/1611.01726/) (2016)
- [When Autonomous Intelligent Goodware will Fight Autonomous Intelligent Malware: A Possible Future of Cyber Defense](https://mlsanity.com/paper/1912.01959/) (2019)
- [Language Models for Novelty Detection in System Call Traces](https://mlsanity.com/paper/2309.02206/) (2023)

## Built on by (most cited first, 8 of 8)

- [LAMD: Context-driven Android Malware Detection and Classification with LLMs](https://mlsanity.com/paper/2502.13055/) (2025)
- [A Contemporary Survey of Large Language Model Assisted Program Analysis](https://mlsanity.com/paper/2502.18474/) (2025)
- [Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluation](https://mlsanity.com/paper/2503.04302/) (2025)
- [REx86: A Local Large Language Model for Assisting in x86 Assembly Reverse Engineering](https://mlsanity.com/paper/2510.20975/) (2025)
- [LoRA-based Parameter-Efficient LLMs for Continuous Learning in Edge-based Malware Detection](https://mlsanity.com/paper/2602.11655/) (2026)
- [LLM-Based Identification of Infostealer Infection Vectors from Screenshots: The Case of Aurora](https://mlsanity.com/paper/2507.23611/) (2025)
- [BEACON: Behavioral Malware Classification with Large Language Model Embeddings and Deep Learning](https://mlsanity.com/paper/2509.14519/) (2025)
- [SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding](https://mlsanity.com/paper/2603.08424/) (2026)

## Abstract (arXiv)

In the current cybersecurity landscape, protecting military devices such as communication and battlefield management systems against sophisticated cyber attacks is crucial. Malware exploits vulnerabilities through stealth methods, often evading traditional detection mechanisms such as software signatures. The application of ML/DL in vulnerability detection has been extensively explored in the literature. However, current ML/DL vulnerability detection methods struggle with understanding the context and intent behind complex attacks. Integrating large language models (LLMs) with system call analysis offers a promising approach to enhance malware detection. This work presents a novel framework leveraging LLMs to classify malware based on system call data. The framework uses transfer learning to adapt pre-trained LLMs for malware detection. By retraining LLMs on a dataset of benign and malicious system calls, the models are refined to detect signs of malware activity. Experiments with a dataset of over 1TB of system calls demonstrate that models with larger context sizes, such as BigBird and Longformer, achieve superior accuracy and F1-Score of approximately 0.86. The results highlight the importance of context size in improving detection rates and underscore the trade-offs between computational complexity and performance. This approach shows significant potential for real-time detection in high-stakes environments, offering a robust solution to evolving cyber threats.