# A Multi-Task Deep Learning Approach for Sensor-based Human Activity Recognition and Segmentation

arXiv: https://arxiv.org/abs/2303.11100 · HTML: https://mlsanity.com/paper/2303.11100/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 75 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 11 of 11)

- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [Ensembles of Deep LSTM Learners for Activity Recognition using Wearables](https://mlsanity.com/paper/1703.09370/) (2017)
- [UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphones](https://mlsanity.com/paper/1611.07688/) (2016)
- [Deep Auto-Set: A Deep Auto-Encoder-Set Network for Activity Recognition Using Wearables](https://mlsanity.com/paper/1811.08127/) (2018)
- [Sensor Data Augmentation by Resampling for Contrastive Learning in Human Activity Recognition](https://mlsanity.com/paper/2109.02054/) (2021)
- [Negative Selection by Clustering for Contrastive Learning in Human Activity Recognition](https://mlsanity.com/paper/2203.12230/) (2022)

## Built on by (most cited first, 4 of 4)

- [HARMamba: Efficient and Lightweight Wearable Sensor Human Activity Recognition Based on Bidirectional Mamba](https://mlsanity.com/paper/2403.20183/) (2024)
- [P2LHAP:Wearable sensor-based human activity recognition, segmentation and forecast through Patch-to-Label Seq2Seq Transformer](https://mlsanity.com/paper/2403.08214/) (2024)
- [Cross-user activity recognition using deep domain adaptation with temporal relation information](https://mlsanity.com/paper/2403.15424/) (2024)
- [Predicting Multiple Clinical Outcomes Related to Functional Recovery and Social Isolation Among Older Adults After Lower-Limb Fracture or Hip Replacement](https://mlsanity.com/paper/2608.23531/) (2026)

## Abstract (arXiv)

Sensor-based human activity segmentation and recognition are two important and challenging problems in many real-world applications and they have drawn increasing attention from the deep learning community in recent years. Most of the existing deep learning works were designed based on pre-segmented sensor streams and they have treated activity segmentation and recognition as two separate tasks. In practice, performing data stream segmentation is very challenging. We believe that both activity segmentation and recognition may convey unique information which can complement each other to improve the performance of the two tasks. In this paper, we firstly proposes a new multitask deep neural network to solve the two tasks simultaneously. The proposed neural network adopts selective convolution and features multiscale windows to segment activities of long or short time durations. First, multiple windows of different scales are generated to center on each unit of the feature sequence. Then, the model is trained to predict, for each window, the activity class and the offset to the true activity boundaries. Finally, overlapping windows are filtered out by non-maximum suppression, and adjacent windows of the same activity are concatenated to complete the segmentation task. Extensive experiments were conducted on eight popular benchmarking datasets, and the results show that our proposed method outperforms the state-of-the-art methods both for activity recognition and segmentation.