# Feature Distribution Shift Mitigation with Contrastive Pretraining for Intrusion Detection

arXiv: https://arxiv.org/abs/2404.15382 · HTML: https://mlsanity.com/paper/2404.15382/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 7 of 7)

- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Vision-and-Language Pretrained Models: A Survey](https://mlsanity.com/paper/2204.07356/) (2022)
- [An Embedding Learning Framework for Numerical Features in CTR Prediction](https://mlsanity.com/paper/2012.08986/) (2020)
- [AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2206.15476/) (2022)

## Built on by (most cited first, 1 of 1)

- [RES-DARE: Failure-Aware Expert Adaptation and Rollback-Safe Self-Repair for Intrusion Detection](https://mlsanity.com/paper/2607.02687/) (2026)

## Abstract (arXiv)

In recent years, there has been a growing interest in using Machine Learning (ML), especially Deep Learning (DL) to solve Network Intrusion Detection (NID) problems. However, the feature distribution shift problem remains a difficulty, because the change in features' distributions over time negatively impacts the model's performance. As one promising solution, model pretraining has emerged as a novel training paradigm, which brings robustness against feature distribution shift and has proven to be successful in Computer Vision (CV) and Natural Language Processing (NLP). To verify whether this paradigm is beneficial for NID problem, we propose SwapCon, a ML model in the context of NID, which compresses shift-invariant feature information during the pretraining stage and refines during the finetuning stage. We exemplify the evidence of feature distribution shift using the Kyoto2006+ dataset. We demonstrate how pretraining a model with the proper size can increase robustness against feature distribution shifts by over 8%. Moreover, we show how an adequate numerical embedding strategy also enhances the performance of pretrained models. Further experiments show that the proposed SwapCon model also outperforms eXtreme Gradient Boosting (XGBoost) and K-Nearest Neighbor (KNN) based models by a large margin.