# Assessing the State of Self-Supervised Human Activity Recognition using Wearables

arXiv: https://arxiv.org/abs/2202.12938 · HTML: https://mlsanity.com/paper/2202.12938/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 99 references that are in the map.
- First posted on arXiv 22 Feb 2022.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [Scaling and Benchmarking Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1905.01235/) (2019)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)

## Built on by (most cited first, 30 of 65)

- [Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data](https://mlsanity.com/paper/2206.02909/) (2022)
- [Pulse-PPG: An Open-Source Field-Trained PPG Foundation Model for Wearable Applications Across Lab and Field Settings](https://mlsanity.com/paper/2502.01108/) (2025)
- [The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models](https://mlsanity.com/paper/2309.06236/) (2023)
- [IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition](https://mlsanity.com/paper/2402.01049/) (2024)
- [RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data](https://mlsanity.com/paper/2411.18822/) (2024)
- [CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking](https://mlsanity.com/paper/2307.16847/) (2023)
- [Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Data](https://mlsanity.com/paper/2403.01133/) (2024)
- [Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task](https://mlsanity.com/paper/2411.14452/) (2024)
- [Transfer Learning in Human Activity Recognition: A Survey](https://mlsanity.com/paper/2401.10185/) (2024)
- [Layout Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST)](https://mlsanity.com/paper/2405.12368/) (2024)
- [Large Language Models are Zero-Shot Recognizers for Activities of Daily Living](https://mlsanity.com/paper/2407.01238/) (2024)
- [Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Them](https://mlsanity.com/paper/2408.12023/) (2024)
- [Combining Public Human Activity Recognition Datasets to Mitigate Labeled Data Scarcity](https://mlsanity.com/paper/2306.13735/) (2023)
- [Cross-Domain HAR: Few Shot Transfer Learning for Human Activity Recognition](https://mlsanity.com/paper/2310.14390/) (2023)
- [Using Large Language Models to Compare Explainable Models for Smart Home Human Activity Recognition](https://mlsanity.com/paper/2408.06352/) (2024)
- [Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognition](https://mlsanity.com/paper/2306.01108/) (2023)
- [Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2211.06173/) (2022)
- [Towards Generalizable Human Activity Recognition: A Survey](https://mlsanity.com/paper/2508.12213/) (2025)
- [Using Self-supervised Learning Can Improve Model Fairness](https://mlsanity.com/paper/2406.02361/) (2024)
- [Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets](https://mlsanity.com/paper/2412.09037/) (2024)
- [Scaling laws in wearable human activity recognition](https://mlsanity.com/paper/2502.03364/) (2025)
- [Scaling Human Activity Recognition: A Comparative Evaluation of Synthetic Data Generation and Augmentation Techniques](https://mlsanity.com/paper/2506.07612/) (2025)
- [AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments](https://mlsanity.com/paper/2506.11773/) (2025)
- [ConvBoost: Boosting ConvNets for Sensor-based Activity Recognition](https://mlsanity.com/paper/2305.13541/) (2023)
- [Representation Learning for Wearable-Based Applications in the Case of Missing Data](https://mlsanity.com/paper/2401.05437/) (2024)
- [Advancing Location-Invariant and Device-Agnostic Motion Activity Recognition on Wearable Devices](https://mlsanity.com/paper/2402.03714/) (2024)
- [Inertia-1: An Open Exploration of Wearable Motion Foundation Models](https://mlsanity.com/paper/2607.06617/) (2026)
- [IMG2IMU: Translating Knowledge from Large-Scale Images to IMU Sensing Applications](https://mlsanity.com/paper/2209.00945/) (2022)
- [Human Activity Recognition Using Self-Supervised Representations of Wearable Data](https://mlsanity.com/paper/2304.14912/) (2023)
- [Don't freeze: Finetune encoders for better Self-Supervised HAR](https://mlsanity.com/paper/2307.01168/) (2023)

## Abstract (arXiv)

The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the field, namely to exploit unlabeled data to derive reliable recognition systems for scenarios where only small amounts of labeled training samples can be collected. As such, self-supervision, i.e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone hand-crafted features for the classic activity recognition chain. Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC, and SimCLR, to name but a few. With the initial success of these methods, the time has come for a systematic inventory and analysis of the potential self-supervised learning has for the field. This paper provides exactly that. We assess the progress of self-supervised HAR research by introducing a framework that performs a multi-faceted exploration of model performance. We organize the framework into three dimensions, each containing three constituent criteria, such that each dimension captures specific aspects of performance, including the robustness to differing source and target conditions, the influence of dataset characteristics, and the feature space characteristics. We utilize this framework to assess seven state-of-the-art self-supervised methods for HAR, leading to the formulation of insights into the properties of these techniques and to establish their value towards learning representations for diverse scenarios.