Finite sample guarantees for quantile estimation: An application to detector threshold tuning

In threshold-based anomaly detection, we want to tune the threshold of a detector to achieve an acceptable false alarm rate. However, tuning the threshold is often a non-trivial task due to unknown detector output distributions. A detector threshold that provides an acceptable false alarm rate is equivalent to a specific quantile of the detector output distribution. Therefore, we use quantile estimators based on order statistics to estimate the detector threshold. The estimation of quantiles from sample data has a more than a century long tradition and we provide three different distribution-free finite sample guarantees for a class of quantile estimators. The first is based on the Dworetzky-Kiefer-Wolfowitz inequality, the second utilizes the Vysochanskij-Petunin inequality, and the third is based on exact confidence intervals for a beta distribution. These guarantees are then compared and used in the detector threshold tuning problem. We use both simulated data as well as data obtained from an experimental setup with the Temperature Control Lab to validate the guarantees provided.