# Transformer-Based Approaches for Sensor-Based Human Activity Recognition: Opportunities and Challenges

arXiv: https://arxiv.org/abs/2410.13605 · HTML: https://mlsanity.com/paper/2410.13605/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Hessian-based Analysis of Large Batch Training and Robustness to Adversaries](https://mlsanity.com/paper/1802.08241/) (2018)
- [Ensembles of Deep LSTM Learners for Activity Recognition using Wearables](https://mlsanity.com/paper/1703.09370/) (2017)
- [Is it Time to Replace CNNs with Transformers for Medical Images?](https://mlsanity.com/paper/2108.09038/) (2021)
- [Relating Adversarially Robust Generalization to Flat Minima](https://mlsanity.com/paper/2104.04448/) (2021)
- [On the Maximum Hessian Eigenvalue and Generalization](https://mlsanity.com/paper/2206.10654/) (2022)
- [ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition](https://mlsanity.com/paper/2308.07815/) (2023)
- [Human Activity Recognition from Wearable Sensor Data Using Self-Attention](https://mlsanity.com/paper/2003.09018/) (2020)
- [UFO-ViT: High Performance Linear Vision Transformer without Softmax](https://mlsanity.com/paper/2109.14382/) (2021)
- [TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation](https://mlsanity.com/paper/2209.09092/) (2022)
- [Empirical Loss Landscape Analysis of Neural Network Activation Functions](https://mlsanity.com/paper/2306.16090/) (2023)

## Built on by (most cited first, 4 of 4)

- [SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognition](https://mlsanity.com/paper/2505.19369/) (2025)
- [A Novel Deep Hybrid Framework with Ensemble-Based Feature Optimization for Robust Real-Time Human Activity Recognition](https://mlsanity.com/paper/2508.18695/) (2025)
- [SOS: A Shuffle Order Strategy for Data Augmentation in Industrial Human Activity Recognition](https://mlsanity.com/paper/2505.10312/) (2025)
- [Hierarchical Motion Captioning Utilizing External Text Data Source](https://mlsanity.com/paper/2509.01471/) (2025)

## Abstract (arXiv)

Transformers have excelled in natural language processing and computer vision, paving their way to sensor-based Human Activity Recognition (HAR). Previous studies show that transformers outperform their counterparts exclusively when they harness abundant data or employ compute-intensive optimization algorithms. However, neither of these scenarios is viable in sensor-based HAR due to the scarcity of data in this field and the frequent need to perform training and inference on resource-constrained devices. Our extensive investigation into various implementations of transformer-based versus non-transformer-based HAR using wearable sensors, encompassing more than 500 experiments, corroborates these concerns. We observe that transformer-based solutions pose higher computational demands, consistently yield inferior performance, and experience significant performance degradation when quantized to accommodate resource-constrained devices. Additionally, transformers demonstrate lower robustness to adversarial attacks, posing a potential threat to user trust in HAR.