# Pre-treatment of outliers and anomalies in plant data: Methodology and case study of a Vacuum Distillation Unit

arXiv: https://arxiv.org/abs/2106.14641 · HTML: https://mlsanity.com/paper/2106.14641/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 50 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 3 of 3)

- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [A review on outlier/anomaly detection in time series data](https://mlsanity.com/paper/2002.04236/) (2020)
- [Change Detection in Multivariate Datastreams: Likelihood and Detectability Loss](https://mlsanity.com/paper/1510.04850/) (2015)

## Built on by (most cited first, 2 of 2)

- [Fast and explainable clustering based on sorting](https://mlsanity.com/paper/2202.01456/) (2022)
- [Machine Learning-Based Soft Sensors for Vacuum Distillation Unit](https://mlsanity.com/paper/2111.11251/) (2021)

## Abstract (arXiv)

Data pre-treatment plays a significant role in improving data quality, thus allowing extraction of accurate information from raw data. One of the data pre-treatment techniques commonly used is outliers detection. The so-called 3σ method is a common practice to identify the outliers. As shown in the manuscript, it does not identify all outliers, resulting in possible distortion of the overall statistics of the data. This problem can have a significant impact on further data analysis and can lead to reduction in the accuracy of predictive models. There is a plethora of various techniques for outliers detection, however, aside from theoretical work, they all require case study work. Two types of outliers were considered: short-term (erroneous data, noise) and long-term outliers (e.g. malfunctioning for longer periods). The data used were taken from the vacuum distillation unit (VDU) of an Asian refinery and included 40 physical sensors (temperature, pressure and flow rate). We used a modified method for 3σ thresholds to identify the short-term outliers, i.e. ensors data are divided into chunks determined by change points and 3σ thresholds are calculated within each chunk representing near-normal distribution. We have shown that piecewise 3σ method offers a better approach to short-term outliers detection than 3σ method applied to the entire time series. Nevertheless, this does not perform well for long-term outliers (which can represent another state in the data). In this case, we used principal component analysis (PCA) with Hotelling's T2 statistics to identify the long-term outliers. The results obtained with PCA were subject to DBSCAN clustering method. The outliers (which were visually obvious and correctly detected by the PCA method) were also correctly identified by DBSCAN which supported the consistency and accuracy of the PCA method.