# Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion

arXiv: https://arxiv.org/abs/2202.04770 · HTML: https://mlsanity.com/paper/2202.04770/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 66 references that are in the map.
- First posted on arXiv 8 Feb 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding](https://mlsanity.com/paper/1802.04431/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [DPGN: Distribution Propagation Graph Network for Few-shot Learning](https://mlsanity.com/paper/2003.14247/) (2020)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art](https://mlsanity.com/paper/2004.00433/) (2020)
- [TimeNet: Pre-trained deep recurrent neural network for time series classification](https://mlsanity.com/paper/1706.08838/) (2017)
- [Intra-Inter Subject Self-supervised Learning for Multivariate Cardiac Signals](https://mlsanity.com/paper/2109.08908/) (2021)
- [Improving Clinical Predictions through Unsupervised Time Series Representation Learning](https://mlsanity.com/paper/1812.00490/) (2018)
- [Time Series Data Augmentation for Neural Networks by Time Warping with a Discriminative Teacher](https://mlsanity.com/paper/2004.08780/) (2020)
- [Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models](https://mlsanity.com/paper/2101.07046/) (2021)

## Built on by (most cited first, 30 of 96)

- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Deep Time Series Models: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2407.13278/) (2024)
- [TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series](https://mlsanity.com/paper/2308.08241/) (2023)
- [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2306.09364/) (2023)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [TSLANet: Rethinking Transformers for Time Series Representation Learning](https://mlsanity.com/paper/2404.08472/) (2024)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting](https://mlsanity.com/paper/2403.09898/) (2024)
- [Bridging the Domain Gap for Multi-Agent Perception](https://mlsanity.com/paper/2210.08451/) (2022)
- [Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series](https://mlsanity.com/paper/2310.14017/) (2023)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer](https://mlsanity.com/paper/2205.09928/) (2022)
- [A Survey on Deep Learning based Time Series Analysis with Frequency Transformation](https://mlsanity.com/paper/2302.02173/) (2023)
- [Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification](https://mlsanity.com/paper/2208.06616/) (2022)
- [TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis](https://mlsanity.com/paper/2402.16412/) (2024)
- [Pulse-PPG: An Open-Source Field-Trained PPG Foundation Model for Wearable Applications Across Lab and Field Settings](https://mlsanity.com/paper/2502.01108/) (2025)
- [Contrastive Learning for Unsupervised Domain Adaptation of Time Series](https://mlsanity.com/paper/2206.06243/) (2022)
- [Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model](https://mlsanity.com/paper/2502.10707/) (2025)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning](https://mlsanity.com/paper/2402.04852/) (2024)
- [Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals](https://mlsanity.com/paper/2309.05927/) (2023)
- [REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning](https://mlsanity.com/paper/2311.00519/) (2023)
- [NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time-Series Pretraining](https://mlsanity.com/paper/2310.07402/) (2023)
- [Label-efficient Time Series Representation Learning: A Review](https://mlsanity.com/paper/2302.06433/) (2023)
- [Individual and Structural Graph Information Bottlenecks for Out-of-Distribution Generalization](https://mlsanity.com/paper/2306.15902/) (2023)
- [Unsupervised Representation Learning for Time Series: A Review](https://mlsanity.com/paper/2308.01578/) (2023)

## Abstract (arXiv)

Unsupervised/self-supervised time series representation learning is a challenging problem because of its complex dynamics and sparse annotations. Existing works mainly adopt the framework of contrastive learning with the time-based augmentation techniques to sample positives and negatives for contrastive training. Nevertheless, they mostly use segment-level augmentation derived from time slicing, which may bring about sampling bias and incorrect optimization with false negatives due to the loss of global context. Besides, they all pay no attention to incorporate the spectral information in feature representation. In this paper, we propose a unified framework, namely Bilinear Temporal-Spectral Fusion (BTSF). Specifically, we firstly utilize the instance-level augmentation with a simple dropout on the entire time series for maximally capturing long-term dependencies. We devise a novel iterative bilinear temporal-spectral fusion to explicitly encode the affinities of abundant time-frequency pairs, and iteratively refines representations in a fusion-and-squeeze manner with Spectrum-to-Time (S2T) and Time-to-Spectrum (T2S) Aggregation modules. We firstly conducts downstream evaluations on three major tasks for time series including classification, forecasting and anomaly detection. Experimental results shows that our BTSF consistently significantly outperforms the state-of-the-art methods.