# Lightweight Transformer in Federated Setting for Human Activity Recognition

arXiv: https://arxiv.org/abs/2110.00244 · HTML: https://mlsanity.com/paper/2110.00244/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables](https://mlsanity.com/paper/1604.08880/) (2016)
- [DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing](https://mlsanity.com/paper/1611.01942/) (2016)
- [Comparative Study of CNN and RNN for Natural Language Processing](https://mlsanity.com/paper/1702.01923/) (2017)
- [FedSel: Federated SGD under Local Differential Privacy with Top-k Dimension Selection](https://mlsanity.com/paper/2003.10637/) (2020)
- [Differential Privacy: An Economic Method for Choosing Epsilon](https://mlsanity.com/paper/1402.3329/) (2014)

## Built on by (most cited first, 2 of 2)

- [Using Anomaly Detection to Detect Poisoning Attacks in Federated Learning Applications](https://mlsanity.com/paper/2207.08486/) (2022)
- [Generative AI Meets Wireless Sensing: Towards Wireless Foundation Model](https://mlsanity.com/paper/2509.15258/) (2025)

## Abstract (arXiv)

Human activity recognition (HAR) is a machine learning task with important applications in healthcare especially in the context of home care of patients and older adults. HAR is often based on data collected from smart sensors, particularly smart home IoT devices such as smartphones, wearables and other body sensors. Deep learning techniques like convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been used for HAR, both in centralized and federated settings. However, these techniques have certain limitations: RNNs cannot be easily parallelized, CNNs have the limitation of sequence length, and both are computationally expensive. Moreover, in home healthcare applications the centralized approach can raise serious privacy concerns since the sensors used by a HAR classifier collect a lot of highly personal and sensitive data about people in the home. In this paper, to address some of such challenges facing HAR, we propose a novel lightweight (one-patch) transformer, which can combine the advantages of RNNs and CNNs without their major limitations, and also TransFed, a more privacy-friendly, federated learning-based HAR classifier using our proposed lightweight transformer. We designed a testbed to construct a new HAR dataset from five recruited human participants, and used the new dataset to evaluate the performance of the proposed HAR classifier in both federated and centralized settings. Additionally, we use another public dataset to evaluate the performance of the proposed HAR classifier in centralized setting to compare it with existing HAR classifiers. The experimental results showed that our proposed new solution outperformed state-of-the-art HAR classifiers based on CNNs and RNNs, whiling being more computationally efficient.