mlsanityData to 14 Oct 2026

SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

arXiv 2402.10198 ↗First posted 15 Feb 2024cs.LG stat.MLcited by 44

Not on the map. Why?

Abstract (arXiv)

Transformer-based architectures achieved breakthrough performance in natural language processing and computer vision, yet they remain inferior to simpler linear baselines in multivariate long-term forecasting. To better understand this phenomenon, we start by studying a toy linear forecasting problem for which we show that transformers are incapable of converging to their true solution despite their high expressive power. We further identify the attention of transformers as being responsible for this low generalization capacity. Building upon this insight, we propose a shallow lightweight transformer model that successfully escapes bad local minima when optimized with sharpness-aware optimization. We empirically demonstrate that this result extends to all commonly used real-world multivariate time series datasets. In particular, SAMformer surpasses current state-of-the-art methods and is on par with the biggest foundation model MOIRAI while having significantly fewer parameters. The code is available at https://github.com/romilbert/samformer.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

34 of 64 references.

References · 34

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  7. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  8. 2020Reformer: The Efficient Transformercited by 1,668
  9. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
  10. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
22 more
  1. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  2. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
  3. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  4. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  5. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  6. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  7. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  8. 2020ResNeSt: Split-Attention Networkscited by 445
  9. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  11. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  12. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  13. 2020Understanding the Difficulty of Training Transformerscited by 210
  14. 2020The Lipschitz Constant of Self-Attentioncited by 174
  15. 2019Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecastingcited by 161
  16. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  17. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  18. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  19. 2023Mimetic Initialization of Self-Attention Layerscited by 54
  20. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 53
  21. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  22. 2021Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Mapscited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 20; 2026: 9.

44 papers.

Sort:

By area:

  1. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 68
  2. 2024Toto: Time Series Optimized Transformer for Observabilitycited by 48
  3. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  4. 2024Scaling-laws for Large Time-series Modelscited by 18
  5. 2024UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecastingcited by 13
  6. 2026EEO-TFV: Escape-Explore Optimizer for Web-Scale Time-Series Forecasting and Vision Analysiscited by 9
  7. 2024Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Modelscited by 8
  8. 2024Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecastingcited by 8
  9. 2025AutoHFormer: Efficient Hierarchical Autoregressive Transformer for Time Series Predictioncited by 8
  10. 2025TS-LIF: A Temporal Segment Spiking Neuron Network for Time Series Forecastingcited by 6
  11. 2025Unlocking Pretrained Vision Transformers for Time Series Classificationcited by 6
  12. 2025How Foundational are Foundation Models for Time Series Forecasting?cited by 5
32 more
  1. 2024User-friendly Foundation Model Adapters for Multivariate Time Series Classificationcited by 3
  2. 2024LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecastingcited by 3
  3. 2024Dataset-Driven Channel Masks in Transformers for Multivariate Time Seriescited by 3
  4. 2025NILMFormer: Non-Intrusive Load Monitoring that Accounts for Non-Stationaritycited by 3
  5. 2025Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecastingcited by 3
  6. 2025IdealTSF: Can Non-Ideal Data Contribute to Enhancing the Performance of Time Series Forecasting Models?cited by 3
  7. 2026Post-Training in Time Series Foundation Models: A Unifying Frameworkcited by 3
  8. 2024Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecastingcited by 2
  9. 2024Training and Evaluating Causal Forecasting Models for Time-Seriescited by 2
  10. 2024PSformer: Parameter-efficient Transformer with Segment Attention for Time Series Forecastingcited by 2
  11. 2025FIC-TSC: Learning Time Series Classification with Fisher Information Constraintcited by 2
  12. 2025Time to Embed: Unlocking Foundation Models for Time Series with Channel Descriptionscited by 2
  13. 2025SKOLR: Structured Koopman Operator Linear RNN for Time-Series Forecastingcited by 2
  14. 2025Why Cannot Neural Networks Master Extrapolation? Insights from Physical Lawscited by 2
  15. 2026Layer by layer, module by module: Choose both for optimal OOD probing of ViTcited by 2
  16. 2024VE: Modeling Multivariate Time Series Correlation with Variate Embeddingcited by 1
  17. 2025SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecastingcited by 1
  18. 2025Sonnet: Spectral Operator Neural Network for Multivariable Time Series Forecastingcited by 1
  19. 2025Foundation vs. Specialized Models: Evaluating Catastrophic Forgetting in Continual Time Series Forecastingcited by 1
  20. 2026vLinear: A Powerful Linear Model for Multivariate Time Series Forecastingcited by 1
  21. 2024Filtered not Mixed: Stochastic Filtering-Based Online Gating for Mixture of Large Language Modelscited by 0
  22. 2024LATST: Are Transformers Necessarily Complex for Time-Series Forecastingcited by 0
  23. 2025Easing Optimization Paths: a Circuit Perspectivecited by 0
  24. 2025Sentinel: Multi-Patch Transformer with Temporal and Channel Attention for Time Series Forecastingcited by 0
  25. 2025FaCTR: Factorized Channel-Temporal Representation Transformers for Efficient Time Series Forecastingcited by 0
  26. 2025Evaluation of a Foundational Model and Stochastic Models for Forecasting Sporadic or Spiky Production Outages of High-Performance Machine Learning Servicescited by 0
  27. 2025Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Modelscited by 0
  28. 2026Deep Time-Series Forecasting in 10 Years: A Surveycited by 0
  29. 2026Adaptive Oscillatory-State Alignment for Time Series Forecastingcited by 0
  30. 2026SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecastingcited by 0
  31. 2026NanoForecast v0.5: Competitive Time Series Forecasting Through Training Pipeline Optimizationcited by 0
  32. 2026Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attentioncited by 0