# Hierarchical Evaluation Function: A Multi-Metric Approach for Optimizing Demand Forecasting Models

arXiv: https://arxiv.org/abs/2508.13057 · HTML: https://mlsanity.com/paper/2508.13057/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 124 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Builds on (most cited first, 15 of 15)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Principles and Algorithms for Forecasting Groups of Time Series: Locality and Globality](https://mlsanity.com/paper/2008.00444/) (2020)
- [Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology](https://mlsanity.com/paper/1809.03006/) (2018)
- [High-Dimensional Granger Causality Tests with an Application to VIX and News](https://mlsanity.com/paper/1912.06307/) (2019)
- [MCDFN: Supply Chain Demand Forecasting via an Explainable Multi-Channel Data Fusion Network Model](https://mlsanity.com/paper/2405.15598/) (2024)
- [Evolving Benchmark Functions to Compare Evolutionary Algorithms via Genetic Programming](https://mlsanity.com/paper/2403.14146/) (2024)
- [Extended formulation and valid inequalities for the multi-item inventory lot-sizing problem with supplier selection](https://mlsanity.com/paper/2002.09916/) (2020)
- [New Metric Formulas that Include Measurement Errors in Machine Learning for Natural Sciences](https://mlsanity.com/paper/2209.15588/) (2022)
- [Bootstrap prediction regions for daily curves of electricity demand and price using functional data](https://mlsanity.com/paper/2401.11885/) (2024)
- [mshw, a forecasting library to predict short-term electricity demand based on multiple seasonal Holt-Winters](https://mlsanity.com/paper/2402.10982/) (2024)
- [Lexidate: Model Evaluation and Selection with Lexicase](https://mlsanity.com/paper/2406.12006/) (2024)
- [Integrated Water Resource Management in the Segura Hydrographic Basin: An Artificial Intelligence Approach](https://mlsanity.com/paper/2411.13566/) (2024)

## Built on by (most cited first, 2 of 2)

- [Adaptive Automatic Model Selection for Demand Forecasting under Heterogeneous Demand Patterns](https://mlsanity.com/paper/2602.13939/) (2026)
- [Beyond a Universal Forecasting Selector: Demand-Conditioned Model Selection across Demand Patterns and Horizons](https://mlsanity.com/paper/2609.04425/) (2026)

## Abstract (arXiv)

Demand forecasting in competitive, uncertain business environments requires models that can integrate multiple evaluation perspectives rather than being restricted to hyperparameter optimization based on a single metric. This traditional approach tends to prioritize one error indicator, which can bias results when metrics provide contradictory signals. In this context, the Hierarchical Evaluation Function (HEF) is proposed as a multi-metric framework for hyperparameter optimization that integrates explanatory power (R2), sensitivity to extreme errors (RMSE), and average accuracy (MAE). The performance of HEF was assessed using four widely recognized benchmark datasets in the forecasting domain: Walmart, M3, M4, and M5. Prediction models were optimized through Grid Search, Particle Swarm Optimization (PSO), and Optuna, and statistical analyses based on difference-of-proportions tests confirmed that HEF delivers superior results compared to a unimetric reference function, regardless of the optimizer employed, with particular relevance for heterogeneous monthly time series (M3) and highly granular daily demand scenarios (M5). The findings demonstrate that HEF improves stability, generalization, and robustness at low computational cost, consolidating its role as a reliable evaluation framework that enhances model selection, enables more accurate demand forecasts, and supports decision-making in dynamic, competitive business environments.