# TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

arXiv: https://arxiv.org/abs/2303.00320 · HTML: https://mlsanity.com/paper/2303.00320/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 83 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [Benchmarking Multivariate Time Series Classification Algorithms](https://mlsanity.com/paper/2007.13156/) (2020)

## Built on by (most cited first, 30 of 44)

- [Learning to Embed Time Series Patches Independently](https://mlsanity.com/paper/2312.16427/) (2023)
- [BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction](https://mlsanity.com/paper/2312.14871/) (2023)
- [ConvTimeNet: A Deep Hierarchical Fully Convolutional Model for Multivariate Time Series Analysis](https://mlsanity.com/paper/2403.01493/) (2024)
- [Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Models](https://mlsanity.com/paper/2402.01749/) (2024)
- [REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning](https://mlsanity.com/paper/2311.00519/) (2023)
- [Self-supervised Representation Learning From Random Data Projectors](https://mlsanity.com/paper/2310.07756/) (2023)
- [Generative Pretrained Hierarchical Transformer for Time Series Forecasting](https://mlsanity.com/paper/2402.16516/) (2024)
- [Improving Time Series Forecasting via Instance-aware Post-hoc Revision](https://mlsanity.com/paper/2505.23583/) (2025)
- [TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation](https://mlsanity.com/paper/2410.05711/) (2024)
- [Advancing Time Series Classification with Multimodal Language Modeling](https://mlsanity.com/paper/2403.12371/) (2024)
- [Position: Beyond Model-Centric Prediction -- Agentic Time Series Forecasting](https://mlsanity.com/paper/2602.01776/) (2026)
- [HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting](https://mlsanity.com/paper/2401.05012/) (2024)
- [CREMA: A Contrastive Regularized Masked Autoencoder for Robust ECG Diagnostics across Clinical Domains](https://mlsanity.com/paper/2407.07110/) (2024)
- [Paving the way toward foundation models for irregular and unaligned Satellite Image Time Series](https://mlsanity.com/paper/2407.08448/) (2024)
- [On the Importance of Step-wise Embeddings for Heterogeneous Clinical Time-Series](https://mlsanity.com/paper/2311.08902/) (2023)
- [UniCL: A Universal Contrastive Learning Framework for Large Time Series Models](https://mlsanity.com/paper/2405.10597/) (2024)
- [Enhancing Masked Time-Series Modeling via Dropping Patches](https://mlsanity.com/paper/2412.15315/) (2024)
- [Towards Robust and Realistic Human Pose Estimation via WiFi Signals](https://mlsanity.com/paper/2501.09411/) (2025)
- [Multiscale Representation Enhanced Temporal Flow Fusion Model for Long-Term Workload Forecasting](https://mlsanity.com/paper/2407.19697/) (2024)
- [A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications](https://mlsanity.com/paper/2410.08224/) (2024)
- [Self-Supervised Dynamical System Representations for Physiological Time-Series](https://mlsanity.com/paper/2512.00239/) (2025)
- [Cross-Modal Representational Knowledge Distillation for Enhanced Spike-Informed LFP Modeling](https://mlsanity.com/paper/2512.12461/) (2025)
- [United We Pretrain, Divided We Fail! Representation Learning for Time Series by Pretraining on 75 Datasets at Once](https://mlsanity.com/paper/2402.15404/) (2024)
- [InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement](https://mlsanity.com/paper/2601.14968/) (2026)
- [Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach](https://mlsanity.com/paper/2604.06727/) (2026)
- [Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series](https://mlsanity.com/paper/2312.03796/) (2023)
- [Content-aware Balanced Spectrum Encoding in Masked Modeling for Time Series Classification](https://mlsanity.com/paper/2412.13232/) (2024)
- [SYNAPSE: Synergizing an Adapter and Finetuning for High-Fidelity EEG Synthesis from a CLIP-Aligned Encoder](https://mlsanity.com/paper/2511.17547/) (2025)
- [Multimodal Physical Fitness Monitoring (PFM) Framework Based on TimeMAE-PFM in Wearable Scenarios](https://mlsanity.com/paper/2404.15294/) (2024)
- [Efficient Multi-View Fusion and Flexible Adaptation to View Missing in Cardiovascular System Signals](https://mlsanity.com/paper/2406.08930/) (2024)

## Abstract (arXiv)

Learning transferable representations from unlabeled time series is crucial for improving performance in data-scarce classification. Existing self-supervised methods often operate at the point level and rely on unidirectional encoding, leading to low semantic density and a mismatch between pre-training and downstream optimization. In this paper, we propose TimeMAE, a self-supervised framework that reformulates masked modeling for time series via semantic unit elevation and decoupled representation learning. Instead of modeling individual time steps, TimeMAE segments time series into non-overlapping sub-series to form semantically enriched units, enabling more informative masked reconstruction while reducing computational cost. To address the representation discrepancy introduced by masking, we design a decoupled masked autoencoder that separately encodes visible and masked regions, avoiding artificial masked tokens in the main encoder. To guide pre-training, we introduce two complementary objectives: masked codeword classification, which discretizes sub-series semantics via a learned tokenizer and masked representation regression, which aligns continuous representations through a momentum-updated target encoder. Extensive experiments on five datasets demonstrate that TimeMAE outperforms competitive baselines, particularly in label-scarce scenarios and transfer learning scenarios.