# Self-Supervised Vision Transformers for Malware Detection

arXiv: https://arxiv.org/abs/2208.07049 · HTML: https://mlsanity.com/paper/2208.07049/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 74 references that are in the map.
- First posted on arXiv 15 Aug 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery](https://mlsanity.com/paper/1711.09082/) (2017)
- [Deep Neural Network Based Malware Detection Using Two Dimensional Binary Program Features](https://mlsanity.com/paper/1508.03096/) (2015)
- [Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting](https://mlsanity.com/paper/2103.13716/) (2021)
- [An Observational Investigation of Reverse Engineers' Processes](https://mlsanity.com/paper/1912.00317/) (2019)
- [A Comparison of Static, Dynamic, and Hybrid Analysis for Malware Detection](https://mlsanity.com/paper/2203.09938/) (2022)
- [Understanding the Impact of Label Granularity on CNN-based Image Classification](https://mlsanity.com/paper/1901.07012/) (2019)
- [DexRay: A Simple, yet Effective Deep Learning Approach to Android Malware Detection based on Image Representation of Bytecode](https://mlsanity.com/paper/2109.03326/) (2021)
- [Unsupervised Anomaly-based Malware Detection using Hardware Features](https://mlsanity.com/paper/1403.1631/) (2014)
- [Self-Supervision, Remote Sensing and Abstraction: Representation Learning Across 3 Million Locations](https://mlsanity.com/paper/2203.04445/) (2022)
- [A Survey on Smartphones Security: Software Vulnerabilities, Malware, and Attacks](https://mlsanity.com/paper/2001.09406/) (2020)

## Built on by (most cited first, 6 of 6)

- [MORPH: Towards Automated Concept Drift Adaptation for Malware Detection](https://mlsanity.com/paper/2401.12790/) (2024)
- [Accelerating Malware Classification: A Vision Transformer Solution](https://mlsanity.com/paper/2409.19461/) (2024)
- [Signal-Based Malware Classification Using 1D CNNs](https://mlsanity.com/paper/2509.06548/) (2025)
- [Self-Supervised Learning for Android Malware Detection on a Time-Stamped Dataset](https://mlsanity.com/paper/2604.23025/) (2026)
- [ThreatVisionAI: A Hybrid CNN-ViT Framework for Image-Based Malware Classification](https://mlsanity.com/paper/2607.03653/) (2026)
- [Bytes to Threats: Byte-Sequence Transformer For Malware Detection](https://mlsanity.com/paper/2610.18787/) (2026)

## Abstract (arXiv)

Malware detection plays a crucial role in cyber-security with the increase in malware growth and advancements in cyber-attacks. Previously unseen malware which is not determined by security vendors are often used in these attacks and it is becoming inevitable to find a solution that can self-learn from unlabeled sample data. This paper presents SHERLOCK, a self-supervision based deep learning model to detect malware based on the Vision Transformer (ViT) architecture. SHERLOCK is a novel malware detection method which learns unique features to differentiate malware from benign programs with the use of image-based binary representation. Experimental results using 1.2 million Android applications across a hierarchy of 47 types and 696 families, shows that self-supervised learning can achieve an accuracy of 97% for the binary classification of malware which is higher than existing state-of-the-art techniques. Our proposed model is also able to outperform state-of-the-art techniques for multi-class malware classification of types and family with macro-F1 score of .497 and .491 respectively.