# Generic Multi-modal Representation Learning for Network Traffic Analysis

arXiv: https://arxiv.org/abs/2405.02649 · HTML: https://mlsanity.com/paper/2405.02649/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Deep Packet: A Novel Approach For Encrypted Traffic Classification Using Deep Learning](https://mlsanity.com/paper/1709.02656/) (2017)
- [ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification](https://mlsanity.com/paper/2202.06335/) (2022)
- [Deep Learning for Encrypted Traffic Classification: An Overview](https://mlsanity.com/paper/1810.07906/) (2018)
- [New Directions in Automated Traffic Analysis](https://mlsanity.com/paper/2008.02695/) (2020)
- [How to Achieve High Classification Accuracy with Just a Few Labels: A Semi-supervised Approach Using Sampled Packets](https://mlsanity.com/paper/1812.09761/) (2018)
- [Hierarchical Character Embeddings: Learning Phonological and Semantic Representations in Languages of Logographic Origin using Recursive Neural Networks](https://mlsanity.com/paper/1912.09913/) (2019)
- [DANTE: A framework for mining and monitoring darknet traffic](https://mlsanity.com/paper/2003.02575/) (2020)
- [Net2Vec: Deep Learning for the Network](https://mlsanity.com/paper/1705.03881/) (2017)
- [Zooming Into the Darknet: Characterizing Internet Background Radiation and its Structural Changes](https://mlsanity.com/paper/2108.00079/) (2021)

## Built on by (most cited first, 1 of 1)

- [Applying JEPA-Style Predictive Learning to JA4-Derived Network Fingerprints](https://mlsanity.com/paper/2607.08465/) (2026)

## Abstract (arXiv)

Network traffic analysis is fundamental for network management, troubleshooting, and security. Tasks such as traffic classification, anomaly detection, and novelty discovery are fundamental for extracting operational information from network data and measurements. We witness the shift from deep packet inspection and basic machine learning to Deep Learning (DL) approaches where researchers define and test a custom DL architecture designed for each specific problem. We here advocate the need for a general DL architecture flexible enough to solve different traffic analysis tasks. We test this idea by proposing a DL architecture based on generic data adaptation modules, followed by an integration module that summarises the extracted information into a compact and rich intermediate representation (i.e. embeddings). The result is a flexible Multi-modal Autoencoder (MAE) pipeline that can solve different use cases. We demonstrate the architecture with traffic classification (TC) tasks since they allow us to quantitatively compare results with state-of-the-art solutions. However, we argue that the MAE architecture is generic and can be used to learn representations useful in multiple scenarios. On TC, the MAE performs on par or better than alternatives while avoiding cumbersome feature engineering, thus streamlining the adoption of DL solutions for traffic analysis.