# Self-Supervised Human Activity Recognition with Localized Time-Frequency Contrastive Representation Learning

arXiv: https://arxiv.org/abs/2209.00990 · HTML: https://mlsanity.com/paper/2209.00990/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 55 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 24 of 24)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [Exploring Contrastive Learning in Human Activity Recognition for Healthcare](https://mlsanity.com/paper/2011.11542/) (2020)
- [On Sampling Strategies for Neural Network-based Collaborative Filtering](https://mlsanity.com/paper/1706.07881/) (2017)
- [Going Deeper with Contextual CNN for Hyperspectral Image Classification](https://mlsanity.com/paper/1604.03519/) (2016)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances](https://mlsanity.com/paper/2111.00418/) (2021)
- [Protecting Sensory Data against Sensitive Inferences](https://mlsanity.com/paper/1802.07802/) (2018)
- [Federated Self-Supervised Learning of Multi-Sensor Representations for Embedded Intelligence](https://mlsanity.com/paper/2007.13018/) (2020)
- [Improving Deep Learning for HAR with shallow LSTMs](https://mlsanity.com/paper/2108.00702/) (2021)
- [Sense and Learn: Self-Supervision for Omnipresent Sensors](https://mlsanity.com/paper/2009.13233/) (2020)
- [A Discriminative CNN Video Representation for Event Detection](https://mlsanity.com/paper/1411.4006/) (2014)
- [Sensor Data Augmentation by Resampling for Contrastive Learning in Human Activity Recognition](https://mlsanity.com/paper/2109.02054/) (2021)
- [Negative Selection by Clustering for Contrastive Learning in Human Activity Recognition](https://mlsanity.com/paper/2203.12230/) (2022)
- [Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion](https://mlsanity.com/paper/2010.13713/) (2020)
- [Human Activity Recognition using Continuous Wavelet Transform and Convolutional Neural Networks](https://mlsanity.com/paper/2106.12666/) (2021)
- [Wearable-based Classification of Running Styles with Deep Learning](https://mlsanity.com/paper/2109.00594/) (2021)

## Built on by (most cited first, 4 of 4)

- [Learning Mutual Excitation for Hand-to-Hand and Human-to-Human Interaction Recognition](https://mlsanity.com/paper/2402.02431/) (2024)
- [Cross-Model Cross-Stream Learning for Self-Supervised Human Action Recognition](https://mlsanity.com/paper/2307.07791/) (2023)
- [Enhancing Human Action Recognition and Violence Detection Through Deep Learning Audiovisual Fusion](https://mlsanity.com/paper/2408.02033/) (2024)
- [Task-Oriented Low-Label Semantic Communication With Self-Supervised Learning](https://mlsanity.com/paper/2505.19940/) (2025)

## Abstract (arXiv)

In this paper, we propose a self-supervised learning solution for human activity recognition with smartphone accelerometer data. We aim to develop a model that learns strong representations from accelerometer signals, in order to perform robust human activity classification, while reducing the model's reliance on class labels. Specifically, we intend to enable cross-dataset transfer learning such that our network pre-trained on a particular dataset can perform effective activity classification on other datasets (successive to a small amount of fine-tuning). To tackle this problem, we design our solution with the intention of learning as much information from the accelerometer signals as possible. As a result, we design two separate pipelines, one that learns the data in time-frequency domain, and the other in time-domain alone. In order to address the issues mentioned above in regards to cross-dataset transfer learning, we use self-supervised contrastive learning to train each of these streams. Next, each stream is fine-tuned for final classification, and eventually the two are fused to provide the final results. We evaluate the performance of the proposed solution on three datasets, namely MotionSense, HAPT, and HHAR, and demonstrate that our solution outperforms prior works in this field. We further evaluate the performance of the method in learning generalized features, by using MobiAct dataset for pre-training and the remaining three datasets for the downstream classification task, and show that the proposed solution achieves better performance in comparison with other self-supervised methods in cross-dataset transfer learning.