# Large Scale Time-Series Representation Learning via Simultaneous Low and High Frequency Feature Bootstrapping

arXiv: https://arxiv.org/abs/2204.11291 · HTML: https://mlsanity.com/paper/2204.11291/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 24 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Sequence-to-Sequence Contrastive Learning for Text Recognition](https://mlsanity.com/paper/2012.10873/) (2020)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data](https://mlsanity.com/paper/2102.06073/) (2021)
- [DEVDAN: Deep Evolving Denoising Autoencoder](https://mlsanity.com/paper/1910.04062/) (2019)
- [Improving the Efficiency of Transformers for Resource-Constrained Devices](https://mlsanity.com/paper/2106.16006/) (2021)
- [Surface Type Classification for Autonomous Robot Indoor Navigation](https://mlsanity.com/paper/1905.00252/) (2019)

## Built on by (most cited first, 5 of 5)

- [S4Sleep: Elucidating the design space of deep-learning-based sleep stage classification models](https://mlsanity.com/paper/2310.06715/) (2023)
- [TopoCL: Topological Contrastive Learning for Time Series](https://mlsanity.com/paper/2502.02924/) (2025)
- [OTCXR: Rethinking Self-supervised Alignment using Optimal Transport for Chest X-ray Analysis](https://mlsanity.com/paper/2404.11868/) (2024)
- [Harmonized Spatial and Spectral Learning for Robust and Generalized Medical Image Segmentation](https://mlsanity.com/paper/2401.10373/) (2024)
- [Self-supervised Activity Representation Learning with Incremental Data: An Empirical Study](https://mlsanity.com/paper/2305.00619/) (2023)

## Abstract (arXiv)

Learning representation from unlabeled time series data is a challenging problem. Most existing self-supervised and unsupervised approaches in the time-series domain do not capture low and high-frequency features at the same time. Further, some of these methods employ large scale models like transformers or rely on computationally expensive techniques such as contrastive learning. To tackle these problems, we propose a non-contrastive self-supervised learning approach efficiently captures low and high-frequency time-varying features in a cost-effective manner. Our method takes raw time series data as input and creates two different augmented views for two branches of the model, by randomly sampling the augmentations from same family. Following the terminology of BYOL, the two branches are called online and target network which allows bootstrapping of the latent representation. In contrast to BYOL, where a backbone encoder is followed by multilayer perceptron (MLP) heads, the proposed model contains additional temporal convolutional network (TCN) heads. As the augmented views are passed through large kernel convolution blocks of the encoder, the subsequent combination of MLP and TCN enables an effective representation of low as well as high-frequency time-varying features due to the varying receptive fields. The two modules (MLP and TCN) act in a complementary manner. We train an online network where each module learns to predict the outcome of the respective module of target network branch. To demonstrate the robustness of our model we performed extensive experiments and ablation studies on five real-world time-series datasets. Our method achieved state-of-art performance on all five real-world datasets.