# CaSS: A Channel-aware Self-supervised Representation Learning Framework for Multivariate Time Series Classification

arXiv: https://arxiv.org/abs/2203.04298 · HTML: https://mlsanity.com/paper/2203.04298/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020)
- [LSTM Fully Convolutional Networks for Time Series Classification](https://mlsanity.com/paper/1709.05206/) (2017)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [Multi-Scale Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/1603.06995/) (2016)
- [Self-attention for raw optical Satellite Time Series Classification](https://mlsanity.com/paper/1910.10536/) (2019)
- [A Time Series Forest for Classification and Feature Extraction](https://mlsanity.com/paper/1302.2277/) (2013)
- [Gated Transformer Networks for Multivariate Time Series Classification](https://mlsanity.com/paper/2103.14438/) (2021)
- [Autoregressive Convolutional Neural Networks for Asynchronous Time Series](https://mlsanity.com/paper/1703.04122/) (2017)
- [Self-supervised representation learning from electroencephalography signals](https://mlsanity.com/paper/1911.05419/) (2019)
- [Learning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks](https://mlsanity.com/paper/1808.06725/) (2018)
- [Medical Time Series Classification with Hierarchical Attention-based Temporal Convolutional Networks: A Case Study of Myotonic Dystrophy Diagnosis](https://mlsanity.com/paper/1903.11748/) (2019)
- [WATTNet: Learning to Trade FX via Hierarchical Spatio-Temporal Representation of Highly Multivariate Time Series](https://mlsanity.com/paper/1909.10801/) (2019)
- [Memory-Gated Recurrent Networks](https://mlsanity.com/paper/2012.13121/) (2020)

## Built on by (most cited first, 1 of 1)

- [Unsupervised Representation Learning for Time Series: A Review](https://mlsanity.com/paper/2308.01578/) (2023)

## Abstract (arXiv)

Self-supervised representation learning of Multivariate Time Series (MTS) is a challenging task and attracts increasing research interests in recent years. Many previous works focus on the pretext task of self-supervised learning and usually neglect the complex problem of MTS encoding, leading to unpromising results. In this paper, we tackle this challenge from two aspects: encoder and pretext task, and propose a unified channel-aware self-supervised learning framework CaSS. Specifically, we first design a new Transformer-based encoder Channel-aware Transformer (CaT) to capture the complex relationships between different time channels of MTS. Second, we combine two novel pretext tasks Next Trend Prediction (NTP) and Contextual Similarity (CS) for the self-supervised representation learning with our proposed encoder. Extensive experiments are conducted on several commonly used benchmark datasets. The experimental results show that our framework achieves new state-of-the-art comparing with previous self-supervised MTS representation learning methods (up to +7.70% improvement on LSST dataset) and can be well applied to the downstream MTS classification.