# TimeFormer: Transformer with Attention Modulation Empowered by Temporal Characteristics for Time Series Forecasting

arXiv: https://arxiv.org/abs/2510.06680 · HTML: https://mlsanity.com/paper/2510.06680/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 58 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods](https://mlsanity.com/paper/2403.20150/) (2024)
- [TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables](https://mlsanity.com/paper/2402.19072/) (2024)
- [TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis](https://mlsanity.com/paper/2410.16032/) (2024)
- [Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA](https://mlsanity.com/paper/1911.06258/) (2019)
- [Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting](https://mlsanity.com/paper/2402.05956/) (2024)
- [From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection](https://mlsanity.com/paper/2409.17515/) (2024)
- [Multi-Scale Adaptive Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2201.04828/) (2022)
- [Unlocking the Power of LSTM for Long Term Time Series Forecasting](https://mlsanity.com/paper/2408.10006/) (2024)
- [Fully-Connected Spatial-Temporal Graph for Multivariate Time-Series Data](https://mlsanity.com/paper/2309.05305/) (2023)
- [LangTime: A Language-Guided Unified Model for Time Series Forecasting with Proximal Policy Optimization](https://mlsanity.com/paper/2503.08271/) (2025)
- [Graph-Aware Contrasting for Multivariate Time-Series Classification](https://mlsanity.com/paper/2309.05202/) (2023)
- [TFDNet: Time-Frequency Enhanced Decomposed Network for Long-term Time Series Forecasting](https://mlsanity.com/paper/2308.13386/) (2023)
- [CSformer: Combining Channel Independence and Mixing for Robust Multivariate Time Series Forecasting](https://mlsanity.com/paper/2312.06220/) (2023)
- [DisMS-TS: Eliminating Redundant Multi-Scale Features for Time Series Classification](https://mlsanity.com/paper/2507.04600/) (2025)
- [A Distillation-based Future-aware Graph Neural Network for Stock Trend Prediction](https://mlsanity.com/paper/2502.10776/) (2025)
- [A novel weighted approach for time series forecasting based on visibility graph](https://mlsanity.com/paper/2103.13870/) (2021)

## Built on by (most cited first, 1 of 1)

- [Beyond Visual Realism: Toward Reliable Financial Time Series Generation](https://mlsanity.com/paper/2601.12990/) (2026)

## Abstract (arXiv)

Although Transformers excel in natural language processing, their extension to time series forecasting remains challenging due to insufficient consideration of the differences between textual and temporal modalities. In this paper, we develop a novel Transformer architecture designed for time series data, aiming to maximize its representational capacity. We identify two key but often overlooked characteristics of time series: (1) unidirectional influence from the past to the future, and (2) the phenomenon of decaying influence over time. These characteristics are introduced to enhance the attention mechanism of Transformers. We propose TimeFormer, whose core innovation is a self-attention mechanism with two modulation terms (MoSA), designed to capture these temporal priors of time series under the constraints of the Hawkes process and causal masking. Additionally, TimeFormer introduces a framework based on multi-scale and subsequence analysis to capture semantic dependencies at different temporal scales, enriching the temporal dependencies. Extensive experiments conducted on multiple real-world datasets show that TimeFormer significantly outperforms state-of-the-art methods, achieving up to a 7.45% reduction in MSE compared to the best baseline and setting new benchmarks on 94.04% of evaluation metrics. Moreover, we demonstrate that the MoSA mechanism can be broadly applied to enhance the performance of other Transformer-based models.