# Self-Supervised Transformer-based Contrastive Learning for Intrusion Detection Systems

arXiv: https://arxiv.org/abs/2505.08816 · HTML: https://mlsanity.com/paper/2505.08816/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification](https://mlsanity.com/paper/2202.06335/) (2022)
- [Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices](https://mlsanity.com/paper/2306.14263/) (2023)
- [NetGPT: Generative Pretrained Transformer for Network Traffic](https://mlsanity.com/paper/2304.09513/) (2023)
- [Anomal-E: A Self-Supervised Network Intrusion Detection System based on Graph Neural Networks](https://mlsanity.com/paper/2207.06819/) (2022)
- [New Directions in Automated Traffic Analysis](https://mlsanity.com/paper/2008.02695/) (2020)
- [FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2304.14746/) (2023)
- [netFound: Principled Design for Network Foundation Models](https://mlsanity.com/paper/2310.17025/) (2023)
- [Real-time Network Intrusion Detection via Decision Transformers](https://mlsanity.com/paper/2312.07696/) (2023)
- [A new hope for network model generalization](https://mlsanity.com/paper/2207.05843/) (2022)
- [Should I Raise The Red Flag? A comprehensive survey of anomaly scoring methods toward mitigating false alarms](https://mlsanity.com/paper/1904.06646/) (2019)
- [A Method for Network Intrusion Detection Using Flow Sequence and BERT Framework](https://mlsanity.com/paper/2310.17127/) (2023)

## Built on by (most cited first, 2 of 2)

- [CSTS: A Canonical Security Telemetry Substrate for AI-Native Cyber Detection](https://mlsanity.com/paper/2603.23459/) (2026)
- [Contrastive Learning and Correlation Clustering for Sequences of Network Telescope Data](https://mlsanity.com/paper/2606.04733/) (2026)

## Abstract (arXiv)

As the digital landscape becomes more interconnected, the frequency and severity of zero-day attacks, have significantly increased, leading to an urgent need for innovative Intrusion Detection Systems (IDS). Machine Learning-based IDS that learn from the network traffic characteristics and can discern attack patterns from benign traffic offer an advanced solution to traditional signature-based IDS. However, they heavily rely on labeled datasets, and their ability to generalize when encountering unseen traffic patterns remains a challenge. This paper proposes a novel self-supervised contrastive learning approach based on transformer encoders, specifically tailored for generalizable intrusion detection on raw packet sequences. Our proposed learning scheme employs a packet-level data augmentation strategy combined with a transformer-based architecture to extract and generate meaningful representations of traffic flows. Unlike traditional methods reliant on handcrafted statistical features (NetFlow), our approach automatically learns comprehensive packet sequence representations, significantly enhancing performance in anomaly identification tasks and supervised learning for intrusion detection. Our transformer-based framework exhibits better performance in comparison to existing NetFlow self-supervised methods. Specifically, we achieve up to a 3% higher AUC in anomaly detection for intra-dataset evaluation and up to 20% higher AUC scores in inter-dataset evaluation. Moreover, our model provides a strong baseline for supervised intrusion detection with limited labeled data, exhibiting an improvement over self-supervised NetFlow models of up to 1.5% AUC when pretrained and evaluated on the same dataset. Additionally, we show the adaptability of our pretrained model when fine-tuned across different datasets, demonstrating strong performance even when lacking benign data from the target domain.