# Improving DNS Exfiltration Detection via Transformer Pretraining

arXiv: https://arxiv.org/abs/2604.09849 · HTML: https://mlsanity.com/paper/2604.09849/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 10 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 4 of 4)

- [NetGPT: Generative Pretrained Transformer for Network Traffic](https://mlsanity.com/paper/2304.09513/) (2023)
- [FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2304.14746/) (2023)
- [DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification](https://mlsanity.com/paper/2409.09143/) (2024)
- [Information-Based Heavy Hitters for Real-Time DNS Data Exfiltration Detection and Prevention](https://mlsanity.com/paper/2307.02614/) (2023)

## Abstract (arXiv)

We study whether in-domain pretraining of Bidirectional Encoder Representations from Transformer (BERT) model improves subdomain-level detection of exfiltration at low false positive rates. While previous work mostly examines fine-tuned generic Transformers, it does not aim to isolate the effect of pretraining on the downstream task of classification. To address this gap, we develop a controlled pipeline where we freeze operating points on validation and transfer them to the test set, thus enabling clean ablations across different label and pretraining budgets. Our results show significant improvements in the left tail of the Receiver Operating Characteristic (ROC) curve, especially against randomly initialized baseline. Additionally, within pretrained model variants, increasing the number of pretraining steps helps the most when more labeled data are available for fine-tuning.