# Transformer-based Models to Deal with Heterogeneous Environments in Human Activity Recognition

arXiv: https://arxiv.org/abs/2209.11750 · HTML: https://mlsanity.com/paper/2209.11750/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 51 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [Exploring Contrastive Learning in Human Activity Recognition for Healthcare](https://mlsanity.com/paper/2011.11542/) (2020)
- [Protecting Sensory Data against Sensitive Inferences](https://mlsanity.com/paper/1802.07802/) (2018)
- [A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison](https://mlsanity.com/paper/2110.10223/) (2021)
- [Optimizing Transformer for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2011.02266/) (2020)
- [An Overview of Human Activity Recognition Using Wearable Sensors: Healthcare and Artificial Intelligence](https://mlsanity.com/paper/2103.15990/) (2021)

## Built on by (most cited first, 12 of 12)

- [Combining Public Human Activity Recognition Datasets to Mitigate Labeled Data Scarcity](https://mlsanity.com/paper/2306.13735/) (2023)
- [MAC-Gaze: Motion-Aware Continual Calibration for Mobile Gaze Tracking](https://mlsanity.com/paper/2505.22769/) (2025)
- [TSRM: A Lightweight Temporal Feature Encoding Architecture for Time Series Forecasting and Imputation](https://mlsanity.com/paper/2504.18878/) (2025)
- [Lightweight Delivery Detection on Doorbell Cameras](https://mlsanity.com/paper/2305.07812/) (2023)
- [rTsfNet: a DNN model with Multi-head 3D Rotation and Time Series Feature Extraction for IMU-based Human Activity Recognition](https://mlsanity.com/paper/2310.19283/) (2023)
- [Time Series Representation Models](https://mlsanity.com/paper/2405.18165/) (2024)
- [Decomposing the Time Series Forecasting Pipeline: A Modular Approach for Time Series Representation, Information Extraction, and Projection](https://mlsanity.com/paper/2507.05891/) (2025)
- [Quantifying Uncertainty in Machine Learning-Based Pervasive Systems: Application to Human Activity Recognition](https://mlsanity.com/paper/2512.09775/) (2025)
- [Parameter-Efficient Fine-Tuning for HAR: Integrating LoRA and QLoRA into Transformer Models](https://mlsanity.com/paper/2512.17983/) (2025)
- [Collaborative Temporal Feature Generation via Critic-Free Reinforcement Learning for Cross-User Sensor-Based Activity Recognition](https://mlsanity.com/paper/2603.16043/) (2026)
- [Efficient and Adaptive Human Activity Recognition via LLM Backbones](https://mlsanity.com/paper/2605.12019/) (2026)
- [FedMHAR: Federated Multimodal Human Activity Recognition using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2609.33492/) (2026)

## Abstract (arXiv)

Human Activity Recognition (HAR) on mobile devices has been demonstrated to be possible using neural models trained on data collected from the device's inertial measurement units. These models have used Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTMs), Transformers or a combination of these to achieve state-of-the-art results with real-time performance. However, these approaches have not been extensively evaluated in real-world situations where the input data may be different from the training data. This paper highlights the issue of data heterogeneity in machine learning applications and how it can hinder their deployment in pervasive settings. To address this problem, we propose and publicly release the code of two sensor-wise Transformer architectures called HART and MobileHART for Human Activity Recognition Transformer. Our experiments on several publicly available datasets show that these HART architectures outperform previous architectures with fewer floating point operations and parameters than conventional Transformers. The results also show they are more robust to changes in mobile position or device brand and hence better suited for the heterogeneous environments encountered in real-life settings. Finally, the source code has been made publicly available.