# Learning to Embed Time Series Patches Independently

arXiv: https://arxiv.org/abs/2312.16427 · HTML: https://mlsanity.com/paper/2312.16427/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 27 Dec 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)
- [Contrastive Masked Autoencoders are Stronger Vision Learners](https://mlsanity.com/paper/2207.13532/) (2022)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2304.05206/) (2023)
- [Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series](https://mlsanity.com/paper/2203.09270/) (2022)

## Built on by (most cited first, 30 of 35)

- [Are Language Models Actually Useful for Time Series Forecasting?](https://mlsanity.com/paper/2406.16964/) (2024)
- [UniTS: A Unified Multi-Task Time Series Model](https://mlsanity.com/paper/2403.00131/) (2024)
- [Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators](https://mlsanity.com/paper/2401.17548/) (2024)
- [Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer](https://mlsanity.com/paper/2405.17478/) (2024)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning](https://mlsanity.com/paper/2402.04852/) (2024)
- [Intervention-Aware Forecasting: Breaking Historical Limits from a System Perspective](https://mlsanity.com/paper/2405.13522/) (2024)
- [MTSCI: A Conditional Diffusion Model for Multivariate Time Series Consistent Imputation](https://mlsanity.com/paper/2408.05740/) (2024)
- [PhaseFormer: From Patches to Phases for Efficient and Effective Time Series Forecasting](https://mlsanity.com/paper/2510.04134/) (2025)
- [DeformTime: Capturing Variable Dependencies with Deformable Attention for Time Series Forecasting](https://mlsanity.com/paper/2406.07438/) (2024)
- [Sequential Order-Robust Mamba for Time Series Forecasting](https://mlsanity.com/paper/2410.23356/) (2024)
- [Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting](https://mlsanity.com/paper/2502.14704/) (2025)
- [Toward Physics-guided Time Series Embedding](https://mlsanity.com/paper/2410.06651/) (2024)
- [Dataset-Driven Channel Masks in Transformers for Multivariate Time Series](https://mlsanity.com/paper/2410.23222/) (2024)
- [A Multi-Modal Knowledge-Enhanced Framework for Vessel Trajectory Prediction](https://mlsanity.com/paper/2503.21834/) (2025)
- [Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting](https://mlsanity.com/paper/2509.20942/) (2025)
- [EXAONE Finance 1.0: An Attention-free Time Series Foundation Model for Financial Time Series](https://mlsanity.com/paper/2609.04239/) (2026)
- [Partial-Multivariate Model for Forecasting](https://mlsanity.com/paper/2408.09703/) (2024)
- [A New Perspective on Time Series Anomaly Detection: Faster Patch-based Broad Learning System](https://mlsanity.com/paper/2412.05498/) (2024)
- [PreMixer: MLP-Based Pre-training Enhanced MLP-Mixers for Large-scale Traffic Forecasting](https://mlsanity.com/paper/2412.13607/) (2024)
- [EMForecaster: A Deep Learning Framework for Time Series Forecasting in Wireless Networks with Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2504.00120/) (2025)
- [Bridging Distribution Gaps in Time Series Foundation Model Pretraining with Prototype-Guided Normalization](https://mlsanity.com/paper/2504.10900/) (2025)
- [Entropy Guided Dynamic Patch Segmentation for Time Series Transformers](https://mlsanity.com/paper/2509.26157/) (2025)
- [A General Framework for Generative Self-supervised Learning in Non-invasive Estimation of Physiological Parameters Using Photoplethysmography](https://mlsanity.com/paper/2604.22780/) (2026)
- [TSRM: A Lightweight Temporal Feature Encoding Architecture for Time Series Forecasting and Imputation](https://mlsanity.com/paper/2504.18878/) (2025)
- [TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting](https://mlsanity.com/paper/2512.22550/) (2025)
- [Multivariate Long-term Time Series Forecasting with Fourier Neural Filter](https://mlsanity.com/paper/2506.09174/) (2025)
- [Decomposing the Time Series Forecasting Pipeline: A Modular Approach for Time Series Representation, Information Extraction, and Projection](https://mlsanity.com/paper/2507.05891/) (2025)
- [DeCoP: Enhancing Self-Supervised Time Series Representation with Dependency Controlled Pre-training](https://mlsanity.com/paper/2509.14642/) (2025)
- [Learning More with Less: A Generalizable, Self-Supervised Framework for Privacy-Preserving Capacity Estimation with EV Charging Data](https://mlsanity.com/paper/2510.05172/) (2025)

## Abstract (arXiv)

Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series. Inspired by masked image modeling in computer vision, recent works first patchify and partially mask out time series, and then train Transformers to capture the dependencies between patches by predicting masked patches from unmasked patches. However, we argue that capturing such patch dependencies might not be an optimal strategy for time series representation learning; rather, learning to embed patches independently results in better time series representations. Specifically, we propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wise MLP that embeds each patch independently. In addition, we introduce complementary contrastive learning to hierarchically capture adjacent time series information efficiently. Our proposed method improves time series forecasting and classification performance compared to state-of-the-art Transformer-based models, while it is more efficient in terms of the number of parameters and training/inference time. Code is available at this repository: https://github.com/seunghan96/pits.