# SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

arXiv: https://arxiv.org/abs/2402.10198 · HTML: https://mlsanity.com/paper/2402.10198/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 64 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 30 of 34)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022)
- [Linear attention is (maybe) all you need (to understand transformer optimization)](https://mlsanity.com/paper/2310.01082/) (2023)

## Built on by (most cited first, 30 of 44)

- [This Time is Different: An Observability Perspective on Time Series Foundation Models](https://mlsanity.com/paper/2505.14766/) (2025)
- [Toto: Time Series Optimized Transformer for Observability](https://mlsanity.com/paper/2407.07874/) (2024)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [Scaling-laws for Large Time-series Models](https://mlsanity.com/paper/2405.13867/) (2024)
- [UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2406.04975/) (2024)
- [EEO-TFV: Escape-Explore Optimizer for Web-Scale Time-Series Forecasting and Vision Analysis](https://mlsanity.com/paper/2602.02551/) (2026)
- [Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models](https://mlsanity.com/paper/2406.04320/) (2024)
- [Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting](https://mlsanity.com/paper/2406.10327/) (2024)
- [AutoHFormer: Efficient Hierarchical Autoregressive Transformer for Time Series Prediction](https://mlsanity.com/paper/2506.16001/) (2025)
- [TS-LIF: A Temporal Segment Spiking Neuron Network for Time Series Forecasting](https://mlsanity.com/paper/2503.05108/) (2025)
- [Unlocking Pretrained Vision Transformers for Time Series Classification](https://mlsanity.com/paper/2506.08641/) (2025)
- [How Foundational are Foundation Models for Time Series Forecasting?](https://mlsanity.com/paper/2510.00742/) (2025)
- [User-friendly Foundation Model Adapters for Multivariate Time Series Classification](https://mlsanity.com/paper/2409.12264/) (2024)
- [LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting](https://mlsanity.com/paper/2410.17159/) (2024)
- [Dataset-Driven Channel Masks in Transformers for Multivariate Time Series](https://mlsanity.com/paper/2410.23222/) (2024)
- [NILMFormer: Non-Intrusive Load Monitoring that Accounts for Non-Stationarity](https://mlsanity.com/paper/2506.05880/) (2025)
- [Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting](https://mlsanity.com/paper/2509.20942/) (2025)
- [IdealTSF: Can Non-Ideal Data Contribute to Enhancing the Performance of Time Series Forecasting Models?](https://mlsanity.com/paper/2512.05442/) (2025)
- [Post-Training in Time Series Foundation Models: A Unifying Framework](https://mlsanity.com/paper/2607.20002/) (2026)
- [Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2410.02168/) (2024)
- [Training and Evaluating Causal Forecasting Models for Time-Series](https://mlsanity.com/paper/2411.00126/) (2024)
- [PSformer: Parameter-efficient Transformer with Segment Attention for Time Series Forecasting](https://mlsanity.com/paper/2411.01419/) (2024)
- [FIC-TSC: Learning Time Series Classification with Fisher Information Constraint](https://mlsanity.com/paper/2505.06114/) (2025)
- [Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptions](https://mlsanity.com/paper/2505.14543/) (2025)
- [SKOLR: Structured Koopman Operator Linear RNN for Time-Series Forecasting](https://mlsanity.com/paper/2506.14113/) (2025)
- [Why Cannot Neural Networks Master Extrapolation? Insights from Physical Laws](https://mlsanity.com/paper/2510.04102/) (2025)
- [Layer by layer, module by module: Choose both for optimal OOD probing of ViT](https://mlsanity.com/paper/2603.05280/) (2026)
- [VE: Modeling Multivariate Time Series Correlation with Variate Embedding](https://mlsanity.com/paper/2409.06169/) (2024)
- [SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2505.02655/) (2025)
- [Sonnet: Spectral Operator Neural Network for Multivariable Time Series Forecasting](https://mlsanity.com/paper/2505.15312/) (2025)

## Abstract (arXiv)

Transformer-based architectures achieved breakthrough performance in natural language processing and computer vision, yet they remain inferior to simpler linear baselines in multivariate long-term forecasting. To better understand this phenomenon, we start by studying a toy linear forecasting problem for which we show that transformers are incapable of converging to their true solution despite their high expressive power. We further identify the attention of transformers as being responsible for this low generalization capacity. Building upon this insight, we propose a shallow lightweight transformer model that successfully escapes bad local minima when optimized with sharpness-aware optimization. We empirically demonstrate that this result extends to all commonly used real-world multivariate time series datasets. In particular, SAMformer surpasses current state-of-the-art methods and is on par with the biggest foundation model MOIRAI while having significantly fewer parameters. The code is available at https://github.com/romilbert/samformer.