# P2LHAP:Wearable sensor-based human activity recognition, segmentation and forecast through Patch-to-Label Seq2Seq Transformer

arXiv: https://arxiv.org/abs/2403.08214 · HTML: https://mlsanity.com/paper/2403.08214/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 73 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [DOC: Deep Open Classification of Text Documents](https://mlsanity.com/paper/1709.08716/) (2017)
- [UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphones](https://mlsanity.com/paper/1611.07688/) (2016)
- [TTS-GAN: A Transformer-based Time-Series Generative Adversarial Network](https://mlsanity.com/paper/2202.02691/) (2022)
- [On Attention Models for Human Activity Recognition](https://mlsanity.com/paper/1805.07648/) (2018)
- [Improving Deep Learning for HAR with shallow LSTMs](https://mlsanity.com/paper/2108.00702/) (2021)
- [Human Activity Recognition Based on Wearable Sensor Data: A Standardization of the State-of-the-Art](https://mlsanity.com/paper/1806.05226/) (2018)
- [Deep Auto-Set: A Deep Auto-Encoder-Set Network for Activity Recognition Using Wearables](https://mlsanity.com/paper/1811.08127/) (2018)
- [Conditional-UNet: A Condition-aware Deep Model for Coherent Human Activity Recognition From Wearables](https://mlsanity.com/paper/2004.09376/) (2020)
- [A Multi-Task Deep Learning Approach for Sensor-based Human Activity Recognition and Segmentation](https://mlsanity.com/paper/2303.11100/) (2023)

## Built on by (most cited first, 4 of 4)

- [A Network Science Approach to Granular Time Series Segmentation](https://mlsanity.com/paper/2505.17640/) (2025)
- [CorVS+: Correspondence-Driven Association of Video Trajectories and Sensors for Identity-Aware Person Localization in Warehouses](https://mlsanity.com/paper/2510.26369/) (2025)
- [Leveraging Imperfect Medical Data: A Manifold-Consistent Spatio-Temporal Network for Sensor-based Human Activity Recognition](https://mlsanity.com/paper/2605.00913/) (2026)
- [Robust Real-Time Monocular Human Pose Estimation with Insole-Based Sensor Fusion](https://mlsanity.com/paper/2610.19494/) (2026)

## Abstract (arXiv)

Traditional deep learning methods struggle to simultaneously segment, recognize, and forecast human activities from sensor data. This limits their usefulness in many fields such as healthcare and assisted living, where real-time understanding of ongoing and upcoming activities is crucial. This paper introduces P2LHAP, a novel Patch-to-Label Seq2Seq framework that tackles all three tasks in a efficient single-task model. P2LHAP divides sensor data streams into a sequence of "patches", served as input tokens, and outputs a sequence of patch-level activity labels including the predicted future activities. A unique smoothing technique based on surrounding patch labels, is proposed to identify activity boundaries accurately. Additionally, P2LHAP learns patch-level representation by sensor signal channel-independent Transformer encoders and decoders. All channels share embedding and Transformer weights across all sequences. Evaluated on three public datasets, P2LHAP significantly outperforms the state-of-the-art in all three tasks, demonstrating its effectiveness and potential for real-world applications.