mlsanityData to 14 Oct 2026

PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting

arXiv 2508.13773 ↗First posted 19 Aug 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite advances in the Transformer architecture, their effectiveness for long-term time series forecasting (LTSF) remains controversial. In this paper, we investigate the potential of integrating explicit periodicity modeling into the self-attention mechanism to enhance the performance of Transformer-based architectures for LTSF. Specifically, we propose PENGUIN, a simple yet effective periodic-nested group attention mechanism. Our approach introduces a periodic-aware relative attention bias to directly capture periodic structures and a grouped multi-query attention mechanism to handle multiple coexisting periodicities (e.g., daily and weekly cycles) within time series data. Extensive experiments across diverse benchmarks demonstrate that PENGUIN consistently outperforms both MLP-based and Transformer-based models. Code is available at https://github.com/ysygMhdxw/AISTATS2026_PENGUIN.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 65 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

36 of 65 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  10. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  11. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  12. 2019Root Mean Square Layer Normalizationcited by 1,480
24 more
  1. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  5. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  6. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  7. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  8. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  9. 2022Transformers in Time Series: A Surveycited by 497
  10. 2019GMAN: A Graph Multi-Attention Network for Traffic Predictioncited by 434
  11. 2023FITS: Modeling Time Series with 10k Parameterscited by 195
  12. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  13. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 141
  14. 2023Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysiscited by 117
  15. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 116
  16. 2024SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameterscited by 107
  17. 2024AutoTimes: Autoregressive Time Series Forecasters via Large Language Modelscited by 65
  18. 2024EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Modelcited by 52
  19. 2023CycleNet: Rethinking Cycle Consistency in Text-Guided Diffusion for Image Manipulationcited by 41
  20. 2024TimeBridge: Non-Stationarity Matters for Long-term Time Series Forecastingcited by 40
  21. 2018DeepTravel: a Neural Network Based Travel Time Estimation Model with Auxiliary Supervisioncited by 21
  22. 2022RNTrajRec: Road Network Enhanced Trajectory Recovery with Spatial-Temporal Transformercited by 21
  23. 2024DGCformer: Deep Graph Clustering Transformer for Multivariate Time Series Forecastingcited by 4
  24. 2025Learning Spatio-Temporal Dynamics for Trajectory Recovery via Time-Aware Transformercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topologycited by 5
  2. 2025Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modelingcited by 0
  3. 2026Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecastingcited by 0