mlsanityData to 7 Oct 2026

Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts

arXiv 2304.09836 ↗First posted 19 Apr 2023cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Multivariate probabilistic time series forecasts are commonly evaluated via proper scoring rules, i.e., functions that are minimal in expectation for the ground-truth distribution. However, this property is not sufficient to guarantee good discrimination in the non-asymptotic regime. In this paper, we provide the first systematic finite-sample study of proper scoring rules for time-series forecasting evaluation. Through a power analysis, we identify the "region of reliability" of a scoring rule, i.e., the set of practical conditions where it can be relied on to identify forecasting errors. We carry out our analysis on a comprehensive synthetic benchmark, specifically designed to test several key discrepancies between ground-truth and forecast distributions, and we gauge the generalizability of our findings to real-world tasks with an application to an electricity production problem. Our results reveal critical shortcomings in the evaluation of multivariate probabilistic forecasts as commonly performed in the literature.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 10 of its 42 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

10 of 42 references.

References · 10

  1. 2021CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputationcited by 577
  2. 2021Monash Time Series Forecasting Archivecited by 256
  3. 2019High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processescited by 147
  4. 2020Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flowscited by 142
  5. 2022TACTiS: Transformer-Attentional Copulas for Time Seriescited by 38
  6. 2021Temporal Latent Auto-Encoder: A Method for Probabilistic Multivariate Time Series Forecastingcited by 32
  7. 2021Evaluating the Discrimination Ability of Proper Multivariate Scoring Rulescited by 8
  8. 2019Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rulescited by 7
  9. 2022Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Abilitycited by 7
  10. 2021A Look at the Evaluation Setup of the M5 Forecasting Competitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 2.

7 papers.

  1. 2023TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Seriescited by 25
  2. 2025From Samples to Scenarios: A New Paradigm for Probabilistic Forecastingcited by 8
  3. 2024Multivariate Probabilistic Time Series Forecasting with Correlated Errorscited by 6
  4. 2025Online Multivariate Regularized Distributional Regression for High-dimensional Probabilistic Electricity Price Forecastingcited by 2
  5. 2024Marginalization Consistent Probabilistic Forecasting of Irregular Time Series via Mixture of Separable flowscited by 0
  6. 2026Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecastingcited by 0
  7. 2026Gaussian Mixture Copula Processes for Irregular Time Seriescited by 0