# NetFlowGen: Leveraging Generative Pre-training for Network Traffic Dynamics

arXiv: https://arxiv.org/abs/2412.20635 · HTML: https://mlsanity.com/paper/2412.20635/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 89 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection](https://mlsanity.com/paper/1802.09089/) (2018)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification](https://mlsanity.com/paper/2202.06335/) (2022)
- [NetGPT: Generative Pretrained Transformer for Network Traffic](https://mlsanity.com/paper/2304.09513/) (2023)
- [Flow-based Network Traffic Generation using Generative Adversarial Networks](https://mlsanity.com/paper/1810.07795/) (2018)
- [Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing](https://mlsanity.com/paper/2110.15534/) (2021)
- [Automating Botnet Detection with Graph Neural Networks](https://mlsanity.com/paper/2003.06344/) (2020)
- [Machine Learning DDoS Detection for Consumer Internet of Things Devices](https://mlsanity.com/paper/1804.04159/) (2018)
- [Autonomous Unknown-Application Filtering and Labeling for DL-based Traffic Classifier Update](https://mlsanity.com/paper/2002.06359/) (2020)
- [Neural Packet Classification](https://mlsanity.com/paper/1902.10319/) (2019)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization](https://mlsanity.com/paper/2410.21411/) (2024)
- [A new hope for network model generalization](https://mlsanity.com/paper/2207.05843/) (2022)
- [SAFE: A Neural Survival Analysis Model for Fraud Early Detection](https://mlsanity.com/paper/1809.04683/) (2018)

## Built on by (most cited first, 2 of 2)

- [An LLM-based Agentic Framework for Accessible Network Control](https://mlsanity.com/paper/2509.20600/) (2025)
- [Generalizable IoT Traffic Representations for Cross-Network Device Identification](https://mlsanity.com/paper/2601.19315/) (2026)

## Abstract (arXiv)

Understanding the traffic dynamics in networks is a core capability for automated systems to monitor and analyze networking behaviors, reducing expensive human efforts and economic risks through tasks such as traffic classification, congestion prediction, and attack detection. However, it is still challenging to accurately model network traffic with machine learning approaches in an efficient and broadly applicable manner. Task-specific models trained from scratch are used for different networking applications, which limits the efficiency of model development and generalization of model deployment. Furthermore, while networking data is abundant, high-quality task-specific labels are often insufficient for training individual models. Large-scale self-supervised learning on unlabeled data provides a natural pathway for tackling these challenges. We propose to pre-train a general-purpose machine learning model to capture traffic dynamics with only traffic data from NetFlow records, with the goal of fine-tuning for different downstream tasks with small amount of labels. Our presented NetFlowGen framework goes beyond a proof-of-concept for network traffic pre-training and addresses specific challenges such as unifying network feature representations, learning from large unlabeled traffic data volume, and testing on real downstream tasks in DDoS attack detection. Experiments demonstrate promising results of our pre-training framework on capturing traffic dynamics and adapting to different networking tasks.