mlsanityData to 7 Oct 2026

Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking

arXiv 2608.20005 ↗First posted 20 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Pretraining time series foundation models across heterogeneous datasets necessitates effective handling of varying sampling frequencies. Current methods either employ dataset-specific patch sizes and separate FFNs, leading to fragmented representations, or enforce a fixed patch size that neglects inherent temporal variations. To address this, we propose SATS, featuring a scale-aware token alignment mechanism that treats patch size as an explicit notion of scale. By incorporating a contrastive-inspired alignment regularizer, SATS aligns representation spaces across scales while preserving distinct modeling capacities. Furthermore, a hybrid masking strategy combining random and contiguous masking is introduced to capture multi-scale temporal structures. Experimental results on LSTF benchmarks demonstrate that SATS achieves a 9.2% improvement in MSE and an 8.3% gain in GIFT-Eval MASE compared to competitive baselines. Notably, SATS consistently delivers SOTA performance while achieving a 65.6% increase in model efficiency over advanced baselines, highlighting its effectiveness and scalability in time series pretraining.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

46 of 54 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
34 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  4. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  5. 2019Root Mean Square Layer Normalizationcited by 1,425
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  8. 2024Chronos: Learning the Language of Time Seriescited by 712
  9. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  10. 2023A decoder-only foundation model for time-series forecastingcited by 650
  11. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  12. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  13. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  14. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  15. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 342
  16. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 333
  17. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  18. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  19. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 188
  20. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 170
  21. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 163
  22. 2022CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecastingcited by 143
  23. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 120
  24. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 102
  25. 2024Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecastingcited by 102
  26. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 88
  27. 2024VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasterscited by 87
  28. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 83
  29. 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 35
  30. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 30
  31. 2025LightGTS: A Lightweight General Time Series Forecasting Modelcited by 28
  32. 2025TimeMosaic: Temporal Heterogeneity Guided Time Series Forecasting via Adaptive Granularity Patch and Segment-wise Decodingcited by 12
  33. 2024ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformercited by 10
  34. 2014Aggregation of predictors for nonstationary sub-linear processes and online adaptive forecasting of time varying autoregressive processescited by 3

Built on by

Not cited yet.