# Timestamp-supervised Wearable-based Activity Segmentation and Recognition with Contrastive Learning and Order-Preserving Optimal Transport

arXiv: https://arxiv.org/abs/2310.09114 · HTML: https://mlsanity.com/paper/2310.09114/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [Alleviating Over-segmentation Errors by Detecting Action Boundaries](https://mlsanity.com/paper/2007.06866/) (2020)
- [Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast](https://mlsanity.com/paper/2110.07110/) (2021)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing](https://mlsanity.com/paper/1611.01942/) (2016)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.09653/) (2022)
- [Ensembles of Deep LSTM Learners for Activity Recognition using Wearables](https://mlsanity.com/paper/1703.09370/) (2017)
- [Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding](https://mlsanity.com/paper/2011.14097/) (2020)
- [Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation](https://mlsanity.com/paper/2209.00638/) (2022)
- [Weakly Supervised Action Labeling in Videos Under Ordering Constraints](https://mlsanity.com/paper/1407.1208/) (2014)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [On Attention Models for Human Activity Recognition](https://mlsanity.com/paper/1805.07648/) (2018)
- [Fast Weakly Supervised Action Segmentation Using Mutual Consistency](https://mlsanity.com/paper/1904.03116/) (2019)
- [Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation](https://mlsanity.com/paper/2008.09524/) (2020)
- [Timestamp-Supervised Action Segmentation with Graph Convolutional Networks](https://mlsanity.com/paper/2206.15031/) (2022)
- [Multi-modal Fusion for Single-Stage Continuous Gesture Recognition](https://mlsanity.com/paper/2011.04945/) (2020)
- [MARS: Mixed Virtual and Real Wearable Sensors for Human Activity Recognition with Multi-Domain Deep Learning Model](https://mlsanity.com/paper/2009.09404/) (2020)
- [Conditional-UNet: A Condition-aware Deep Model for Coherent Human Activity Recognition From Wearables](https://mlsanity.com/paper/2004.09376/) (2020)
- [Invariant Feature Learning for Sensor-based Human Activity Recognition](https://mlsanity.com/paper/2012.07963/) (2020)

## Built on by (most cited first, 10 of 10)

- [EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling](https://mlsanity.com/paper/2412.10235/) (2024)
- [RadarLLM: Empowering Large Language Models to Understand Human Motion from Millimeter-Wave Point Cloud Sequence](https://mlsanity.com/paper/2504.09862/) (2025)
- [mmDEAR: mmWave Point Cloud Density Enhancement for Accurate Human Body Reconstruction](https://mlsanity.com/paper/2503.02375/) (2025)
- [SMART: Scene-motion-aware human action recognition framework for mental disorder group](https://mlsanity.com/paper/2406.04649/) (2024)
- [Suite-IN: Aggregating Motion Features from Apple Suite for Robust Inertial Navigation](https://mlsanity.com/paper/2411.07828/) (2024)
- [Learning Satellite Pattern-of-Life Identification: A Diffusion-based Approach](https://mlsanity.com/paper/2412.10814/) (2024)
- [Suite-IN++: A FlexiWear BodyNet Integrating Global and Local Motion Features from Apple Suite for Robust Inertial Navigation](https://mlsanity.com/paper/2504.00438/) (2025)
- [DAP: Doppler-aware Point Network for Heterogeneous mmWave Action Recognition](https://mlsanity.com/paper/2605.09604/) (2026)
- [Diverse Yet Consistent: Context-Guided Diffusion with Energy-Based Joint Refinement for Multi-Agent Motion Prediction](https://mlsanity.com/paper/2605.22017/) (2026)
- [PSEE: Progressive Sensor Event Expansion for Point-Supervised Temporal Action Localization](https://mlsanity.com/paper/2609.21462/) (2026)

## Abstract (arXiv)

Human activity recognition (HAR) with wearables is one of the serviceable technologies in ubiquitous and mobile computing applications. The sliding-window scheme is widely adopted while suffering from the multi-class windows problem. As a result, there is a growing focus on joint segmentation and recognition with deep-learning methods, aiming at simultaneously dealing with HAR and time-series segmentation issues. However, obtaining the full activity annotations of wearable data sequences is resource-intensive or time-consuming, while unsupervised methods yield poor performance. To address these challenges, we propose a novel method for joint activity segmentation and recognition with timestamp supervision, in which only a single annotated sample is needed in each activity segment. However, the limited information of sparse annotations exacerbates the gap between recognition and segmentation tasks, leading to sub-optimal model performance. Therefore, the prototypes are estimated by class-activation maps to form a sample-to-prototype contrast module for well-structured embeddings. Moreover, with the optimal transport theory, our approach generates the sample-level pseudo-labels that take advantage of unlabeled data between timestamp annotations for further performance improvement. Comprehensive experiments on four public HAR datasets demonstrate that our model trained with timestamp supervision is superior to the state-of-the-art weakly-supervised methods and achieves comparable performance to the fully-supervised approaches.