mlsanityData to 7 Oct 2026

A Transformer-based Framework for Multivariate Time Series Representation Learning

arXiv 2010.02803 ↗First posted 6 Oct 2020cs.LG cs.AIcited by 449

Not on the map. Why?

Abstract (arXiv)

In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series. Pre-trained models can be potentially used for downstream tasks such as regression and classification, forecasting and missing value imputation. By evaluating our models on several benchmark datasets for multivariate time series regression and classification, we show that not only does our modeling approach represent the most successful method employing unsupervised learning of multivariate time series presented to date, but also that it exceeds the current state-of-the-art performance of supervised methods; it does so even when the number of training samples is very limited, while offering computational efficiency. Finally, we demonstrate that unsupervised pre-training of our transformer models offers a substantial performance benefit over fully supervised learning, even without leveraging additional unlabeled data, i.e., by reusing the same data samples through the unsupervised objective.

Key facts. Cited by 449 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 54; 2023: 99; 2024: 129; 2025: 95; 2026: 60.

449 papers.

Sort:

By area:

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  2. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  3. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  4. 2022Transformers in Time Series: A Surveycited by 489
  5. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
  6. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  7. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  8. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  9. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  10. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  11. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 165
  12. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 146
38 more
  1. 2023A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detectioncited by 145
  2. 2022CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecastingcited by 143
  3. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 120
  4. 2022Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecastingcited by 117
  5. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  6. 2021Graph-Guided Network for Irregularly Sampled Multivariate Time Seriescited by 110
  7. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  8. 2021Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Seriescited by 96
  9. 2022Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusioncited by 96
  10. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 93
  11. 2024VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasterscited by 87
  12. 2023Time Series as Images: Vision Transformer for Irregularly Sampled Time Seriescited by 79
  13. 2023Non-autoregressive Conditional Diffusion Models for Time Series Predictioncited by 77
  14. 2022Hierarchical Graph Transformer with Adaptive Node Samplingcited by 72
  15. 2022Flowformer: Linearizing Transformers with Conservation Flowscited by 69
  16. 2023Deep Learning for Time Series Classification and Extrinsic Regression: A Current Surveycited by 68
  17. 2024Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathologycited by 66
  18. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 63
  19. 2023Ti-MAE: Self-Supervised Masked Time Series Autoencoderscited by 63
  20. 2024TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecastingcited by 63
  21. 2024ContiFormer: Continuous-Time Transformer for Irregular Time Series Modelingcited by 61
  22. 2023A Unified, Scalable Framework for Neural Population Decodingcited by 59
  23. 2022Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecastingcited by 54
  24. 2023Time Series Contrastive Learning with Information-Aware Augmentationscited by 52
  25. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  26. 2023TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learningcited by 51
  27. 2023DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Predictioncited by 49
  28. 2023Soft Contrastive Learning for Time Seriescited by 49
  29. 2024EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputscited by 48
  30. 2022Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformercited by 47
  31. 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
  32. 2023Improving Position Encoding of Transformers for Multivariate Time Series Classificationcited by 44
  33. 2023TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoderscited by 42
  34. 2023Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Datacited by 40
  35. 2023Learning to Embed Time Series Patches Independentlycited by 34
  36. 2022Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modelingcited by 33
  37. 2024TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Modelcited by 33
  38. 2024A Survey of Deep Learning and Foundation Models for Time Series Forecastingcited by 32

50 most cited of 449 All 449 (JSON)