# Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts

arXiv: https://arxiv.org/abs/2304.09836 · HTML: https://mlsanity.com/paper/2304.09836/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes](https://mlsanity.com/paper/1910.03002/) (2019)
- [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020)
- [TACTiS: Transformer-Attentional Copulas for Time Series](https://mlsanity.com/paper/2202.03528/) (2022)
- [Temporal Latent Auto-Encoder: A Method for Probabilistic Multivariate Time Series Forecasting](https://mlsanity.com/paper/2101.10460/) (2021)
- [Evaluating the Discrimination Ability of Proper Multivariate Scoring Rules](https://mlsanity.com/paper/2101.12693/) (2021)
- [Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules](https://mlsanity.com/paper/1910.07325/) (2019)
- [Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability](https://mlsanity.com/paper/2201.08671/) (2022)
- [A Look at the Evaluation Setup of the M5 Forecasting Competition](https://mlsanity.com/paper/2108.03588/) (2021)

## Built on by (most cited first, 7 of 7)

- [TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series](https://mlsanity.com/paper/2310.01327/) (2023)
- [From Samples to Scenarios: A New Paradigm for Probabilistic Forecasting](https://mlsanity.com/paper/2509.19975/) (2025)
- [Multivariate Probabilistic Time Series Forecasting with Correlated Errors](https://mlsanity.com/paper/2402.01000/) (2024)
- [Online Multivariate Regularized Distributional Regression for High-dimensional Probabilistic Electricity Price Forecasting](https://mlsanity.com/paper/2504.02518/) (2025)
- [Marginalization Consistent Probabilistic Forecasting of Irregular Time Series via Mixture of Separable flows](https://mlsanity.com/paper/2406.07246/) (2024)
- [Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting](https://mlsanity.com/paper/2602.10182/) (2026)
- [Gaussian Mixture Copula Processes for Irregular Time Series](https://mlsanity.com/paper/2605.23632/) (2026)

## Abstract (arXiv)

Multivariate probabilistic time series forecasts are commonly evaluated via proper scoring rules, i.e., functions that are minimal in expectation for the ground-truth distribution. However, this property is not sufficient to guarantee good discrimination in the non-asymptotic regime. In this paper, we provide the first systematic finite-sample study of proper scoring rules for time-series forecasting evaluation. Through a power analysis, we identify the "region of reliability" of a scoring rule, i.e., the set of practical conditions where it can be relied on to identify forecasting errors. We carry out our analysis on a comprehensive synthetic benchmark, specifically designed to test several key discrepancies between ground-truth and forecast distributions, and we gauge the generalizability of our findings to real-world tasks with an application to an electricity production problem. Our results reveal critical shortcomings in the evaluation of multivariate probabilistic forecasts as commonly performed in the literature.