mlsanityData to 14 Oct 2026

Self-Distilled Representation Learning for Time Series

arXiv 2311.11335 ↗First posted 19 Nov 2023cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning for time-series data holds potential similar to that recently unleashed in Natural Language Processing and Computer Vision. While most existing works in this area focus on contrastive learning, we propose a conceptually simple yet powerful non-contrastive approach, based on the data2vec self-distillation framework. The core of our method is a student-teacher scheme that predicts the latent representation of an input time series from masked views of the same time series. This strategy avoids strong modality-specific assumptions and biases typically introduced by the design of contrastive sample pairs. We demonstrate the competitiveness of our approach for classification and forecasting as downstream tasks, comparing with state-of-the-art self-supervised learning methods on the UCR and UEA archives as well as the ETT and Electricity datasets.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 19 Nov 2023.

Builds on

31 of 37 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  11. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
19 more
  1. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  2. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  4. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  5. 2018Deep learning for time series classification: a reviewcited by 576
  6. 2022Transformers in Time Series: A Surveycited by 497
  7. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 471
  8. 2020Time Series Forecasting With Deep Learning: A Surveycited by 459
  9. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 453
  10. 2018The UCR Time Series Archivecited by 431
  11. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 345
  12. 2018The UEA multivariate time series classification archive, 2018cited by 338
  13. 2019Unsupervised Scalable Representation Learning for Multivariate Time Seriescited by 269
  14. 2021Unsupervised Representation Learning for Time Series with Temporal Neighborhood Codingcited by 224
  15. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  16. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 99
  17. 2023Ti-MAE: Self-Supervised Masked Time Series Autoencoderscited by 63
  18. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  19. 2017TimeNet: Pre-trained deep recurrent neural network for time series classificationcited by 46

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025AimTS: Augmented Series and Image Contrastive Learning for Time Series Classificationcited by 6
  2. 2025Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Datacited by 1
  3. 2025Learning Time-Series Representations by Hierarchical Uniformity-Tolerance Latent Balancingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification
  2. 2024Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask
  3. 2024TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation
  4. 2026Enhancing deep learning models for time series classification via knowledge distillation
  5. 2024Blending Low and High-Level Semantics of Time Series for Better Masked Time Series Generation
  6. 2024HiMTM: Hierarchical Multi-Scale Masked Time Series Modeling with Self-Distillation for Long-Term Forecasting
  7. 2026WinoTS: Wavelet-based Self-Distillation for Time Series Models
  8. 2021Self-Distilled Self-Supervised Representation Learning