# TgDLF2.0: Theory-guided deep-learning for electrical load forecasting via Transformer and transfer learning

arXiv: https://arxiv.org/abs/2210.02448 · HTML: https://mlsanity.com/paper/2210.02448/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data](https://mlsanity.com/paper/1612.08544/) (2016)
- [Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling](https://mlsanity.com/paper/1710.11431/) (2017)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [PCAN: 3D Attention Map Learning Using Contextual Information for Point Cloud Based Retrieval](https://mlsanity.com/paper/1904.09793/) (2019)
- [FLAT: Chinese NER Using Flat-Lattice Transformer](https://mlsanity.com/paper/2004.11795/) (2020)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [Deep Learning of Subsurface Flow via Theory-guided Neural Network](https://mlsanity.com/paper/1911.00103/) (2019)
- [Theory-guided hard constraint projection (HCP): a knowledge-based data-driven scientific machine learning method](https://mlsanity.com/paper/2012.06148/) (2020)
- [LightSeq2: Accelerated Training for Transformer-based Models on GPUs](https://mlsanity.com/paper/2110.05722/) (2021)
- [Integration of knowledge and data in machine learning](https://mlsanity.com/paper/2202.10337/) (2022)
- [Understanding Multi-Head Attention in Abstractive Summarization](https://mlsanity.com/paper/1911.03898/) (2019)
- [Ensemble Neural Networks (ENN): A gradient-free stochastic method](https://mlsanity.com/paper/1908.01113/) (2019)
- [Ensemble long short-term memory (EnLSTM) network](https://mlsanity.com/paper/2004.13562/) (2020)
- [Deep Learning of Dynamic Subsurface Flow via Theory-guided Generative Adversarial Network](https://mlsanity.com/paper/2006.13305/) (2020)

## Built on by (most cited first, 2 of 2)

- [Transformer Training Strategies for Forecasting Multiple Load Time Series](https://mlsanity.com/paper/2306.10891/) (2023)
- [Domain adaption and physical constrains transfer learning for shale gas production](https://mlsanity.com/paper/2312.10920/) (2023)

## Abstract (arXiv)

Electrical energy is essential in today's society. Accurate electrical load forecasting is beneficial for better scheduling of electricity generation and saving electrical energy. In this paper, we propose theory-guided deep-learning load forecasting 2.0 (TgDLF2.0) to solve this issue, which is an improved version of the theory-guided deep-learning framework for load forecasting via ensemble long short-term memory (TgDLF). TgDLF2.0 introduces the deep-learning model Transformer and transfer learning on the basis of dividing the electrical load into dimensionless trends and local fluctuations, which realizes the utilization of domain knowledge, captures the long-term dependency of the load series, and is more appropriate for realistic scenarios with scarce samples. Cross-validation experiments on different districts show that TgDLF2.0 is approximately 16% more accurate than TgDLF and saves more than half of the training time. TgDLF2.0 with 50% weather noise has the same accuracy as TgDLF without noise, which proves its robustness. We also preliminarily mine the interpretability of Transformer in TgDLF2.0, which may provide future potential for better theory guidance. Furthermore, experiments demonstrate that transfer learning can accelerate convergence of the model in half the number of training epochs and achieve better performance.