# Sampling Streaming Data with Parallel Vector Quantization -- PVQ

arXiv: https://arxiv.org/abs/2210.01792 · HTML: https://mlsanity.com/paper/2210.01792/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 64 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Estimating Continuous Distributions in Bayesian Classifiers](https://mlsanity.com/paper/1302.4964/) (2013)
- [Petuum: A New Platform for Distributed Machine Learning on Big Data](https://mlsanity.com/paper/1312.7651/) (2013)
- [Benchmark and Survey of Automated Machine Learning Frameworks](https://mlsanity.com/paper/1904.12054/) (2019)
- [Efficient Machine Learning for Big Data: A Review](https://mlsanity.com/paper/1503.05296/) (2015)
- [A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment](https://mlsanity.com/paper/1810.07748/) (2018)
- [Evaluation of Machine Learning-based Anomaly Detection Algorithms on an Industrial Modbus/TCP Data Set](https://mlsanity.com/paper/1905.11757/) (2019)

## Built on by (most cited first, 1 of 1)

- [An efficient and straightforward online quantization method for a data stream through remove-birth updating](https://mlsanity.com/paper/2306.12574/) (2023)

## Abstract (arXiv)

Accumulation of corporate data in the cloud has attracted more enterprise applications to the cloud creating data gravity. As a consequence, network traffic has become more cloud centric. This increase in cloud centric traffic poses new challenges in designing learning systems for streaming data due to class imbalance. The number of classes plays a vital role in the accuracy of the classifiers built from the data streams. In this paper, we present a vector quantization-based sampling method, which substantially reduces the class imbalance in data streams. We demonstrate its effectiveness by conducting experiments on network traffic and anomaly dataset with commonly used ML model building methods; Multilayered Perceptron on TensorFlow backend, Support Vector Machines, K-Nearest Neighbour, and Random Forests. We built models using parallel processing, batch processing, and randomly selecting samples. We show that the accuracy of classification models improves when the data streams are pre-processed with our method. We used out of the box hyper-parameters of these classifiers and auto sklearn for hyperparameter optimization.