# A better measure of relative prediction accuracy for model selection and model estimation

arXiv: https://arxiv.org/abs/2105.05249 · HTML: https://mlsanity.com/paper/2105.05249/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2021.

## Built on by (most cited first, 20 of 20)

- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [Forecasting GPU Performance for Deep Learning Training and Inference](https://mlsanity.com/paper/2407.13853/) (2024)
- [FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning](https://mlsanity.com/paper/2309.10283/) (2023)
- [AGAR a microbial colony dataset for deep learning detection](https://mlsanity.com/paper/2108.01234/) (2021)
- [Measuring the Validity of Clustering Validation Datasets](https://mlsanity.com/paper/2503.01097/) (2025)
- [Sanity Check for External Clustering Validation Benchmarks using Internal Validation Measures](https://mlsanity.com/paper/2209.10042/) (2022)
- [HARDCORE: H-field and power loss estimation for arbitrary waveforms with residual, dilated convolutional neural networks in ferrite cores](https://mlsanity.com/paper/2401.11488/) (2024)
- [Application Performance Modeling via Tensor Completion](https://mlsanity.com/paper/2210.10184/) (2022)
- [Cross-Region Building Counting in Satellite Imagery using Counting Consistency](https://mlsanity.com/paper/2110.13558/) (2021)
- [CAHR-Net: Condition-Adaptive Hysteresis Reconstruction for Compact and Interpretable Magnetic Core Loss Modeling](https://mlsanity.com/paper/2609.01991/) (2026)
- [Modeling Price Elasticity for Occupancy Prediction in Hotel Dynamic Pricing](https://mlsanity.com/paper/2208.03135/) (2022)
- [Few-Shot Load Forecasting Under Data Scarcity in Smart Grids: A Meta-Learning Approach](https://mlsanity.com/paper/2406.05887/) (2024)
- [Construction Cost Index Forecasting: A Multi-feature Fusion Approach](https://mlsanity.com/paper/2108.10155/) (2021)
- [DVS: Deep Visibility Series and its Application in Construction Cost Index Forecasting](https://mlsanity.com/paper/2111.04071/) (2021)
- [Large-Scale Cell-Level Quality of Service Estimation on 5G Networks Using Machine Learning Techniques](https://mlsanity.com/paper/2212.14071/) (2022)
- [Enhancing Forecasting with a 2D Time Series Approach for Cohort-Based Data](https://mlsanity.com/paper/2508.15369/) (2025)
- [Time to Revist Exact Match](https://mlsanity.com/paper/2509.16720/) (2025)
- [Physics-Informed Framework for Impact Identification in Aerospace Composites](https://mlsanity.com/paper/2603.28593/) (2026)
- [Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection](https://mlsanity.com/paper/2604.13882/) (2026)
- [Cross-Material Support Transfer for Core-Loss Prediction Under Waveform Covariate Shift](https://mlsanity.com/paper/2609.31659/) (2026)

## Abstract (arXiv)

Surveys show that the mean absolute percentage error (MAPE) is the most widely used measure of forecast accuracy in businesses and organizations. It is however, biased: When used to select among competing prediction methods it systematically selects those whose predictions are too low. This is not widely discussed and so is not generally known among practitioners. We explain why this happens. We investigate an alternative relative accuracy measure which avoids this bias: the log of the accuracy ratio: log (prediction / actual). Relative accuracy is particularly relevant if the scatter in the data grows as the value of the variable grows (heteroscedasticity). We demonstrate using simulations that for heteroscedastic data (modelled by a multiplicative error factor) the proposed metric is far superior to MAPE for model selection. Another use for accuracy measures is in fitting parameters to prediction models. Minimum MAPE models do not predict a simple statistic and so theoretical analysis is limited. We prove that when the proposed metric is used instead, the resulting least squares regression model predicts the geometric mean. This important property allows its theoretical properties to be understood.