# Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability

arXiv: https://arxiv.org/abs/2201.08671 · HTML: https://mlsanity.com/paper/2201.08671/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 21 Jan 2022.

## Builds on (most cited first, 15 of 15)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2101.12072/) (2021)
- [High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes](https://mlsanity.com/paper/1910.03002/) (2019)
- [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020)
- [Model-Free Renewable Scenario Generation Using Generative Adversarial Networks](https://mlsanity.com/paper/1707.09676/) (2017)
- [Improving Palliative Care with Deep Learning](https://mlsanity.com/paper/1711.06402/) (2017)
- [DeepDownscale: a Deep Learning Strategy for High-Resolution Weather Forecast](https://mlsanity.com/paper/1808.05264/) (2018)
- [Probabilistic Forecasting of Sensory Data with Generative Adversarial Networks - ForGAN](https://mlsanity.com/paper/1903.12549/) (2019)
- [If You Like It, GAN It. Probabilistic Multivariate Times Series Forecast With GAN](https://mlsanity.com/paper/2005.01181/) (2020)
- [CRED: A Deep Residual Network of Convolutional and Recurrent Units for Earthquake Signal Detection](https://mlsanity.com/paper/1810.01965/) (2018)
- [Wind Energy: Forecasting Challenges for Its Operational Management](https://mlsanity.com/paper/1312.6471/) (2013)
- [PhaseLink: A Deep Learning Approach to Seismic Phase Association](https://mlsanity.com/paper/1809.02880/) (2018)
- [Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules](https://mlsanity.com/paper/1910.07325/) (2019)

## Built on by (most cited first, 7 of 7)

- [Modeling Temporal Data as Continuous Functions with Stochastic Process Diffusion](https://mlsanity.com/paper/2211.02590/) (2022)
- [TACTiS: Transformer-Attentional Copulas for Time Series](https://mlsanity.com/paper/2202.03528/) (2022)
- [Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts](https://mlsanity.com/paper/2304.09836/) (2023)
- [Multivariate Probabilistic Time Series Forecasting with Correlated Errors](https://mlsanity.com/paper/2402.01000/) (2024)
- [Recurrent Interpolants for Probabilistic Time Series Prediction](https://mlsanity.com/paper/2409.11684/) (2024)
- [Probabilistic Forecasting of Irregular Time Series via Conditional Flows](https://mlsanity.com/paper/2402.06293/) (2024)
- [Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting](https://mlsanity.com/paper/2602.10182/) (2026)

## Abstract (arXiv)

The recent developments in the machine learning domain have enabled the development of complex multivariate probabilistic forecasting models. Therefore, it is pivotal to have a precise evaluation method to gauge the performance and predictability power of these complex methods. To do so, several evaluation metrics have been proposed in the past (such as Energy Score, Dawid-Sebastiani score, variogram score), however, they cannot reliably measure the performance of a probabilistic forecaster. Recently, CRPS-sum has gained a lot of prominence as a reliable metric for multivariate probabilistic forecasting. This paper presents a systematic evaluation of CRPS-sum to understand its discrimination ability. We show that the statistical properties of target data affect the discrimination ability of CRPS-Sum. Furthermore, we highlight that CRPS-Sum calculation overlooks the performance of the model on each dimension. These flaws can lead us to an incorrect assessment of model performance. Finally, with experiments on the real-world dataset, we demonstrate that the shortcomings of CRPS-Sum provide a misleading indication of the probabilistic forecasting performance method. We show that it is easily possible to have a better CRPS-Sum for a dummy model, which looks like random noise, in comparison to the state-of-the-art method.