# Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017

arXiv: https://arxiv.org/abs/2606.11098 · HTML: https://mlsanity.com/paper/2606.11098/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [A Survey of Network-based Intrusion Detection Data Sets](https://mlsanity.com/paper/1903.02460/) (2019)
- [Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey](https://mlsanity.com/paper/2408.07583/) (2024)
- [Introduction to Sequence Modeling with Transformers](https://mlsanity.com/paper/2502.19597/) (2025)

## Abstract (arXiv)

Recent deep learning approaches for network intrusion detection increasingly incorporate temporal architectures such as recurrent networks and Transformers, often reporting near-perfect performance on CIC-IDS2017. However, many existing studies neither supply their temporal modules with genuine sequence inputs nor evaluate under realistic, leakage-free conditions, making it unclear whether reported gains arise from true sequence-modelling capability. In this work, we reformulate CIC-IDS2017 as a temporal intrusion-detection task by constructing ordered flow sequences from network conversations and benchmarking nine classical and deep learning architectures under a random split, two leakage-free splits, and a padding-scheme ablation. The central finding is that padding convention, not architecture, determines the Transformer's performance: on genuinely sequential (non-padded) windows the Transformer achieves the highest macro-F1 of any model in the experiment (0.89), yet under zero-pad+mask evaluation it drops markedly while LSTM, GRU, and 1D-CNN remain stable and a class-balanced Random Forest is the most robust model overall. The same repeat-last padding cue masks a 67× increase in the Transformer's false-alarm rate that only leakage-free evaluation reveals. Evaluation methodology -- specifically padding convention and split protocol -- thus has a larger effect on reported performance than architectural choice. We advocate for leakage-free splits, explicit padding disclosure, and sequence-aware benchmarking as standard practice in future IDS research. Our code and implementation details are available here: https://github.com/zachmocz/temporal-ids-bench.