# User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detection

arXiv: https://arxiv.org/abs/2506.23446 · HTML: https://mlsanity.com/paper/2506.23446/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [AnoViT: Unsupervised Anomaly Detection and Localization with Vision Transformer-based Encoder-Decoder](https://mlsanity.com/paper/2203.10808/) (2022)
- [Deep Learning for Insider Threat Detection: Review, Challenges and Opportunities](https://mlsanity.com/paper/2005.12433/) (2020)
- [LAN: Learning Adaptive Neighbors for Real-Time Insider Threat Detection](https://mlsanity.com/paper/2403.09209/) (2024)
- [Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research](https://mlsanity.com/paper/2110.08975/) (2021)
- [TransLog: A Unified Transformer-based Framework for Log Anomaly Detection](https://mlsanity.com/paper/2201.00016/) (2021)

## Built on by (most cited first, 1 of 1)

- [OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection](https://mlsanity.com/paper/2603.22499/) (2026)

## Abstract (arXiv)

Insider threat detection presents unique challenges due to the authorized status of malicious actors and the subtlety of anomalous behaviors. Existing machine learning methods often treat user activity as isolated events, thereby failing to leverage sequential dependencies in user behavior. In this study, we propose a User-Based Sequencing (UBS) methodology, transforming the CERT insider threat dataset into structured temporal sequences suitable for deep sequential modeling. We deploy a Transformer Encoder architecture to model benign user activity and employ its reconstruction errors as anomaly scores. These scores are subsequently evaluated using three unsupervised outlier detection algorithms: One-Class SVM (OCSVM), Local Outlier Factor (LOF), and Isolation Forest (iForest). Across four rigorously designed test sets, including combinations of multiple CERT dataset releases, our UBS-Transformer pipeline consistently achieves state-of-the-art performance - notably 96.61% accuracy, 99.43% recall, 96.38% F1-score, 95.00% AUROC, and exceptionally low false negative (0.0057) and false positive (0.0571) rates. Comparative analyses demonstrate that our approach substantially outperforms tabular and conventional autoencoder baselines, underscoring the efficacy of sequential user modeling and advanced anomaly detection in the insider threat domain.