# Multi-Objective Model Selection for Time Series Forecasting

arXiv: https://arxiv.org/abs/2202.08485 · HTML: https://mlsanity.com/paper/2202.08485/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 84 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [A Multi-Horizon Quantile Recurrent Forecaster](https://mlsanity.com/paper/1711.11053/) (2017)
- [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015)
- [High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes](https://mlsanity.com/paper/1910.03002/) (2019)
- [Principles and Algorithms for Forecasting Groups of Time Series: Locality and Globality](https://mlsanity.com/paper/2008.00444/) (2020)
- [Multi-objective Asynchronous Successive Halving](https://mlsanity.com/paper/2106.12639/) (2021)
- [AutoAI-TS: AutoAI for Time Series Forecasting](https://mlsanity.com/paper/2102.12347/) (2021)
- [The Kalai-Smorodinski solution for many-objective Bayesian optimization](https://mlsanity.com/paper/1902.06565/) (2019)
- [A multi-objective perspective on jointly tuning hardware and hyperparameters](https://mlsanity.com/paper/2106.05680/) (2021)
- [Learning Multiple Defaults for Machine Learning Algorithms](https://mlsanity.com/paper/1811.09409/) (2018)
- [Practical and sample efficient zero-shot HPO](https://mlsanity.com/paper/2007.13382/) (2020)

## Built on by (most cited first, 8 of 8)

- [AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2308.05566/) (2023)
- [TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications](https://mlsanity.com/paper/2311.02971/) (2023)
- [Deep Learning based Forecasting: a case study from the online fashion industry](https://mlsanity.com/paper/2305.14406/) (2023)
- [TransNAS-TSAD: Harnessing Transformers for Multi-Objective Neural Architecture Search in Time Series Anomaly Detection](https://mlsanity.com/paper/2311.18061/) (2023)
- [Adaptive Sampling for Probabilistic Forecasting under Distribution Shift](https://mlsanity.com/paper/2302.11870/) (2023)
- [Ensembling Finetuned Language Models for Text Classification](https://mlsanity.com/paper/2410.19889/) (2024)
- [Multi-output Ensembles for Multi-step Forecasting](https://mlsanity.com/paper/2306.14563/) (2023)
- [Hyperparameter Tuning MLPs for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2403.04477/) (2024)

## Abstract (arXiv)

Research on time series forecasting has predominantly focused on developing methods that improve accuracy. However, other criteria such as training time or latency are critical in many real-world applications. We therefore address the question of how to choose an appropriate forecasting model for a given dataset among the plethora of available forecasting methods when accuracy is only one of many criteria. For this, our contributions are two-fold. First, we present a comprehensive benchmark, evaluating 7 classical and 6 deep learning forecasting methods on 44 heterogeneous, publicly available datasets. The benchmark code is open-sourced along with evaluations and forecasts for all methods. These evaluations enable us to answer open questions such as the amount of data required for deep learning models to outperform classical ones. Second, we leverage the benchmark evaluations to learn good defaults that consider multiple objectives such as accuracy and latency. By learning a mapping from forecasting models to performance metrics, we show that our method PARETOSELECT is able to accurately select models from the Pareto front -- alleviating the need to train or evaluate many forecasting models for model selection. To the best of our knowledge, PARETOSELECT constitutes the first method to learn default models in a multi-objective setting.