# Ti-MAE: Self-Supervised Masked Time Series Autoencoders

arXiv: https://arxiv.org/abs/2301.08871 · HTML: https://mlsanity.com/paper/2301.08871/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 21 Jan 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)

## Built on by (most cited first, 30 of 63)

- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series](https://mlsanity.com/paper/2401.03955/) (2024)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2310.08278/) (2023)
- [CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns](https://mlsanity.com/paper/2409.18479/) (2024)
- [ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation](https://mlsanity.com/paper/2312.01728/) (2023)
- [A Survey of Deep Learning and Foundation Models for Time Series Forecasting](https://mlsanity.com/paper/2401.13912/) (2024)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Spatial-Temporal-Decoupled Masked Pre-training for Spatiotemporal Forecasting](https://mlsanity.com/paper/2312.00516/) (2023)
- [Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning](https://mlsanity.com/paper/2402.04852/) (2024)
- [LSM-2: Learning from Incomplete Wearable Sensor Data](https://mlsanity.com/paper/2506.05321/) (2025)
- [REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning](https://mlsanity.com/paper/2311.00519/) (2023)
- [NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time-Series Pretraining](https://mlsanity.com/paper/2310.07402/) (2023)
- [TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling](https://mlsanity.com/paper/2402.02475/) (2024)
- [Merlin: Multi-View Representation Learning for Robust Multivariate Time Series Forecasting with Unfixed Missing Rates](https://mlsanity.com/paper/2506.12459/) (2025)
- [Deep Learning for Time Series Forecasting: A Survey](https://mlsanity.com/paper/2503.10198/) (2025)
- [Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification](https://mlsanity.com/paper/2510.26777/) (2025)
- [HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting](https://mlsanity.com/paper/2401.05012/) (2024)
- [OTIS: Learning High-Quality Time Series Features With Tiny Encoders](https://mlsanity.com/paper/2410.07299/) (2024)
- [SIGMA-PPG: Statistical-prior Informed Generative Masking Architecture for PPG Foundation Model](https://mlsanity.com/paper/2601.21031/) (2026)
- [Reinforcement Learning with Maskable Stock Representation for Portfolio Management in Customizable Stock Pools](https://mlsanity.com/paper/2311.10801/) (2023)
- [Mask the Redundancy: Evolving Masking Representation Learning for Multivariate Time-Series Clustering](https://mlsanity.com/paper/2511.17008/) (2025)
- [UniCL: A Universal Contrastive Learning Framework for Large Time Series Models](https://mlsanity.com/paper/2405.10597/) (2024)
- [A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis](https://mlsanity.com/paper/2308.07301/) (2023)
- [Self-Supervised Learning for Time Series: Contrastive or Generative?](https://mlsanity.com/paper/2403.09809/) (2024)
- [LSTTN: A Long-Short Term Transformer-based Spatio-temporal Neural Network for Traffic Flow Forecasting](https://mlsanity.com/paper/2403.16495/) (2024)
- [Multi-View Contrastive Learning for Robust Domain Adaptation in Medical Time Series Analysis](https://mlsanity.com/paper/2506.22393/) (2025)
- [Self-Distilled Representation Learning for Time Series](https://mlsanity.com/paper/2311.11335/) (2023)
- [ECG Signal Denoising Using Multi-scale Patch Embedding and Transformers](https://mlsanity.com/paper/2407.11065/) (2024)
- [Neural Fourier Modelling: A Highly Compact Approach to Time-Series Analysis](https://mlsanity.com/paper/2410.04703/) (2024)

## Abstract (arXiv)

Multivariate Time Series forecasting has been an increasingly popular topic in various applications and scenarios. Recently, contrastive learning and Transformer-based models have achieved good performance in many long-term series forecasting tasks. However, there are still several issues in existing methods. First, the training paradigm of contrastive learning and downstream prediction tasks are inconsistent, leading to inaccurate prediction results. Second, existing Transformer-based models which resort to similar patterns in historical time series data for predicting future values generally induce severe distribution shift problems, and do not fully leverage the sequence information compared to self-supervised methods. To address these issues, we propose a novel framework named Ti-MAE, in which the input time series are assumed to follow an integrate distribution. In detail, Ti-MAE randomly masks out embedded time series data and learns an autoencoder to reconstruct them at the point-level. Ti-MAE adopts mask modeling (rather than contrastive learning) as the auxiliary task and bridges the connection between existing representation learning and generative Transformer-based methods, reducing the difference between upstream and downstream forecasting tasks while maintaining the utilization of original time series data. Experiments on several public real-world datasets demonstrate that our framework of masked autoencoding could learn strong representations directly from the raw data, yielding better performance in time series forecasting and classification tasks.