# Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules

arXiv: https://arxiv.org/abs/1910.07325 · HTML: https://mlsanity.com/paper/1910.07325/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [Generative machine learning methods for multivariate ensemble post-processing](https://mlsanity.com/paper/2211.01345/) (2022)
- [Random Noise vs State-of-the-Art Probabilistic Forecasting Methods : A Case Study on CRPS-Sum Discrimination Ability](https://mlsanity.com/paper/2201.08671/) (2022)
- [Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts](https://mlsanity.com/paper/2304.09836/) (2023)
- [Efficiently Generating Correlated Sample Paths from Multi-step Time Series Foundation Models](https://mlsanity.com/paper/2510.02224/) (2025)
- [Simulation-based Forecasting for Intraday Power Markets: Modelling Fundamental Drivers for Location, Shape and Scale of the Price Distribution](https://mlsanity.com/paper/2211.13002/) (2022)
- [Generating ensembles of spatially-coherent in-situ forecasts using flow matching](https://mlsanity.com/paper/2504.03463/) (2025)
- [Online Multivariate Regularized Distributional Regression for High-dimensional Probabilistic Electricity Price Forecasting](https://mlsanity.com/paper/2504.02518/) (2025)

## Abstract (arXiv)

In recent years, probabilistic forecasting is an emerging topic, which is why there is a growing need of suitable methods for the evaluation of multivariate predictions. We analyze the sensitivity of the most common scoring rules, especially regarding quality of the forecasted dependency structures. Additionally, we propose scoring rules based on the copula, which uniquely describes the dependency structure for every probability distribution with continuous marginal distributions. Efficient estimation of the considered scoring rules and evaluation methods such as the Diebold-Mariano test are discussed. In detailed simulation studies, we compare the performance of the renowned scoring rules and the ones we propose. Besides extended synthetic studies based on recently published results we also consider a real data example. We find that the energy score, which is probably the most widely used multivariate scoring rule, performs comparably well in detecting forecast errors, also regarding dependencies. This contradicts other studies. The results also show that a proposed copula score provides very strong distinction between models with correct and incorrect dependency structure. We close with a comprehensive discussion on the proposed methodology.