# Is Self-Pretraining really useful to improve diagnosis in medical Time Series?

arXiv: https://arxiv.org/abs/2608.06122 · HTML: https://mlsanity.com/paper/2608.06122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 21 of 21)

- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Are Large-scale Datasets Necessary for Self-Supervised Pre-training?](https://mlsanity.com/paper/2112.10740/) (2021)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [Effectiveness of self-supervised pre-training for speech recognition](https://mlsanity.com/paper/1911.03912/) (2019)
- [Self Pre-training with Masked Autoencoders for Medical Image Classification and Segmentation](https://mlsanity.com/paper/2203.05573/) (2022)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [TimeNet: Pre-trained deep recurrent neural network for time series classification](https://mlsanity.com/paper/1706.08838/) (2017)
- [Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors](https://mlsanity.com/paper/2310.02980/) (2023)
- [UniMTS: Unified Pre-training for Motion Time Series](https://mlsanity.com/paper/2410.19818/) (2024)
- [Large Pre-trained time series models for cross-domain Time series analysis tasks](https://mlsanity.com/paper/2311.11413/) (2023)
- [Multimodal Pretraining of Medical Time Series and Notes](https://mlsanity.com/paper/2312.06855/) (2023)
- [TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling](https://mlsanity.com/paper/2402.02475/) (2024)
- [A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks](https://mlsanity.com/paper/2007.00800/) (2020)
- [Robust Lane Detection through Self Pre-training with Masked Sequential Autoencoders and Fine-tuning with Customized PolyLoss](https://mlsanity.com/paper/2305.17271/) (2023)
- [Self Pre-training with Adaptive Mask Autoencoders for Variable-Contrast 3D Medical Imaging](https://mlsanity.com/paper/2501.09096/) (2025)
- [Masked Autoencoder Self Pre-Training for Defect Detection in Microelectronics](https://mlsanity.com/paper/2504.10021/) (2025)
- [Towards Understanding Self-Pretraining for Sequence Classification](https://mlsanity.com/paper/2605.21070/) (2026)

## Abstract (arXiv)

Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. We evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, and we systematically vary model depth to examine how capacity interacts with pre-training benefits. Across datasets and configurations, SPT consistently improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. These findings indicate that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings.