# Transformer Training Strategies for Forecasting Multiple Load Time Series

arXiv: https://arxiv.org/abs/2306.10891 · HTML: https://mlsanity.com/paper/2306.10891/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 19 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Review of Low Voltage Load Forecasting: Methods, Applications, and Recommendations](https://mlsanity.com/paper/2106.00006/) (2021)
- [Short-Term Electricity Load Forecasting Using the Temporal Fusion Transformer: Effect of Grid Hierarchies and Data Sources](https://mlsanity.com/paper/2305.10559/) (2023)
- [Transfer Learning in Deep Learning Models for Building Load Forecasting: Case of Limited Data](https://mlsanity.com/paper/2301.10663/) (2023)
- [TgDLF2.0: Theory-guided deep-learning for electrical load forecasting via Transformer and transfer learning](https://mlsanity.com/paper/2210.02448/) (2022)

## Built on by (most cited first, 6 of 6)

- [Explainable Load Forecasting with Covariate-Informed Time Series Foundation Models](https://mlsanity.com/paper/2604.28149/) (2026)
- [Thermal-GEMs: Generalized Models for Building Thermal Dynamics](https://mlsanity.com/paper/2604.16443/) (2026)
- [Transfer Learning on Transformers for Building Energy Consumption Forecasting -- A Comparative Study](https://mlsanity.com/paper/2410.14107/) (2024)
- [Explainable time-series forecasting with sampling-free SHAP for Transformers](https://mlsanity.com/paper/2512.20514/) (2025)
- [A Benchmark for Electrical Load Forecasting Across Grid Levels: Time-Series Transformers Outperform Established Methods](https://mlsanity.com/paper/2607.15705/) (2026)
- [HN-MVTS: HyperNetwork-based Multivariate Time Series Forecasting](https://mlsanity.com/paper/2511.08340/) (2025)

## Abstract (arXiv)

In the smart grid of the future, accurate load forecasts on the level of individual clients can help to balance supply and demand locally and to prevent grid outages. While the number of monitored clients will increase with the ongoing smart meter rollout, the amount of data per client will always be limited. We evaluate whether a Transformer load forecasting model benefits from a transfer learning strategy, where a global univariate model is trained on the load time series from multiple clients. In experiments with two datasets containing load time series from several hundred clients, we find that the global training strategy is superior to the multivariate and local training strategies used in related work. On average, the global training strategy results in 21.8% and 12.8% lower forecasting errors than the two other strategies, measured across forecasting horizons from one day to one month into the future. A comparison to linear models, multi-layer perceptrons and LSTMs shows that Transformers are effective for load forecasting when they are trained with the global training strategy.