# TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling

arXiv: https://arxiv.org/abs/2402.02475 · HTML: https://mlsanity.com/paper/2402.02475/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Time Series Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2002.12478/) (2020)
- [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)

## Built on by (most cited first, 20 of 20)

- [Deep Time Series Models: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2407.13278/) (2024)
- [VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2408.17253/) (2024)
- [Trajectory Flow Matching with Applications to Clinical Time Series Modeling](https://mlsanity.com/paper/2410.21154/) (2024)
- [TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation](https://mlsanity.com/paper/2410.05711/) (2024)
- [VisionTS++: Cross-Modal Time Series Foundation Model with Continual Pre-trained Vision Backbones](https://mlsanity.com/paper/2508.04379/) (2025)
- [Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives](https://mlsanity.com/paper/2506.24124/) (2025)
- [Metadata Matters for Time Series: Informative Forecasting with Transformers](https://mlsanity.com/paper/2410.03806/) (2024)
- [Sequential Order-Robust Mamba for Time Series Forecasting](https://mlsanity.com/paper/2410.23356/) (2024)
- [GTM: A General Time-series Model for Enhanced Representation Learning of Time-Series Data](https://mlsanity.com/paper/2502.03264/) (2025)
- [Dataset-Driven Channel Masks in Transformers for Multivariate Time Series](https://mlsanity.com/paper/2410.23222/) (2024)
- [Bridging Distribution Gaps in Time Series Foundation Model Pretraining with Prototype-Guided Normalization](https://mlsanity.com/paper/2504.10900/) (2025)
- [Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generation](https://mlsanity.com/paper/2502.15466/) (2025)
- [Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgment](https://mlsanity.com/paper/2506.01290/) (2025)
- [Enhancing Contrastive Learning-based Electrocardiogram Pretrained Model with Patient Memory Queue](https://mlsanity.com/paper/2506.06310/) (2025)
- [SDSC:A Structure-Aware Metric for Semantic Signal Representation Learning](https://mlsanity.com/paper/2507.14516/) (2025)
- [Physical activities enable scalable foundation modelling for broad-spectrum health prediction](https://mlsanity.com/paper/2607.06954/) (2026)
- [Is Self-Pretraining really useful to improve diagnosis in medical Time Series?](https://mlsanity.com/paper/2608.06122/) (2026)
- [Contrastive Representation-Guided Genetic Minority Oversampling for Imbalanced Time-Series Classification](https://mlsanity.com/paper/2608.22804/) (2026)
- [WinoTS: Wavelet-based Self-Distillation for Time Series Models](https://mlsanity.com/paper/2609.39337/) (2026)
- [CORD: Learning Reusable Degradation Representations Across Heterogeneous Physical Systems](https://mlsanity.com/paper/2609.39784/) (2026)

## Abstract (arXiv)

Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. Prior methods are mainly based on pre-training techniques well-acknowledged in vision or language, such as masked modeling and contrastive learning. However, randomly masking time series or calculating series-wise similarity will distort or neglect inherent temporal correlations crucial in time series data. To emphasize temporal correlation modeling, this paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks. Concretely, TimeSiam pre-trains Siamese encoders to capture intrinsic temporal correlations between randomly sampled past and current subseries. With a simple data augmentation method (e.g.~masking), TimeSiam can benefit from diverse augmented subseries and learn internal time-dependent representations through a past-to-current reconstruction. Moreover, learnable lineage embeddings are also introduced to distinguish temporal distance between sampled series and further foster the learning of diverse temporal correlations. TimeSiam consistently outperforms extensive advanced pre-training baselines, demonstrating superior forecasting and classification capabilities across 13 standard benchmarks in both intra- and cross-domain scenarios.