# Domain Adaptation for Time series Transformers using One-step fine-tuning

arXiv: https://arxiv.org/abs/2401.06524 · HTML: https://mlsanity.com/paper/2401.06524/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Transformers in Time-series Analysis: A Tutorial](https://mlsanity.com/paper/2205.01138/) (2022)
- [Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension](https://mlsanity.com/paper/1911.00202/) (2019)
- [An Investigation of Replay-based Approaches for Continual Learning](https://mlsanity.com/paper/2108.06758/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Survey of AIOps in the Era of Large Language Models](https://mlsanity.com/paper/2507.12472/) (2025)
- [Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting](https://mlsanity.com/paper/2505.17872/) (2025)
- [SwiftQueue: Optimizing Low-Latency Applications with Swift Packet Queuing](https://mlsanity.com/paper/2410.06112/) (2024)

## Abstract (arXiv)

The recent breakthrough of Transformers in deep learning has drawn significant attention of the time series community due to their ability to capture long-range dependencies. However, like other deep learning models, Transformers face limitations in time series prediction, including insufficient temporal understanding, generalization challenges, and data shift issues for the domains with limited data. Additionally, addressing the issue of catastrophic forgetting, where models forget previously learned information when exposed to new data, is another critical aspect that requires attention in enhancing the robustness of Transformers for time series tasks. To address these limitations, in this paper, we pre-train the time series Transformer model on a source domain with sufficient data and fine-tune it on the target domain with limited data. We introduce the One-step fine-tuning approach, adding some percentage of source domain data to the target domains, providing the model with diverse time series instances. We then fine-tune the pre-trained model using a gradual unfreezing technique. This helps enhance the model's performance in time series prediction for domains with limited data. Extensive experimental results on two real-world datasets show that our approach improves over the state-of-the-art baselines by 4.35% and 11.54% for indoor temperature and wind power prediction, respectively.