# HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound

arXiv: https://arxiv.org/abs/2605.02596 · HTML: https://mlsanity.com/paper/2605.02596/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 51 references that are in the map.
- First posted on arXiv 4 May 2026.

## Builds on (most cited first, 11 of 11)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Epic-Sounds: A Large-scale Dataset of Actions That Sound](https://mlsanity.com/paper/2302.00646/) (2023)
- [WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition](https://mlsanity.com/paper/2304.05088/) (2023)
- [Improving Deep Learning for HAR with shallow LSTMs](https://mlsanity.com/paper/2108.00702/) (2021)
- [Hang-Time HAR: A Benchmark Dataset for Basketball Activity Recognition using Wrist-Worn Inertial Sensors](https://mlsanity.com/paper/2305.13124/) (2023)
- [Learning Human Identity from Motion Patterns](https://mlsanity.com/paper/1511.03908/) (2015)
- [MEx: Multi-modal Exercises Dataset for Human Activity Recognition](https://mlsanity.com/paper/1908.08992/) (2019)
- [iWash: A Smartwatch Handwashing Quality Assessment and Reminder System with Real-time Feedback in the Context of Infectious Disease](https://mlsanity.com/paper/2009.10317/) (2020)
- [Multi-modal Atmospheric Sensing to Augment Wearable IMU-Based Hand Washing Detection](https://mlsanity.com/paper/2410.03549/) (2024)
- [WatchHAR: Real-time On-device Human Activity Recognition System for Smartwatches](https://mlsanity.com/paper/2509.04736/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset](https://mlsanity.com/paper/2606.27886/) (2026)

## Abstract (arXiv)

With each sensing modality exhibiting inherent strengths and limitations, multi-modal approaches for wearable Human Activity Recognition (HAR) are becoming increasingly relevant -- particularly for recognizing Activities of Daily Living (ADLs), where individual modalities often produce ambiguous signals for similar or complex activities. This work introduces HARMES, a multi-modal wearable dataset combining three wrist-recorded modalities: motion sensing via an Inertial Measurement Unit (IMU), atmospheric environmental sensors (humidity, temperature, and pressure), and audio. Collected from 20 participants performing household activities in their own homes, HARMES totals over 80 hours of recorded data, with approximately three hours of labeled activity data per participant across 15 ADL classes. To the best of our knowledge, HARMES is the first dataset to combine this particular sensor trio, and it is nearly six times larger than the previously largest wrist-inertial-acoustic HAR dataset. In an extensive benchmark, we evaluate cross-subject generalization and conduct an ablation study revealing that modality contributions are activity-dependent and can provide complementary value, particularly for activities that are ambiguous from motion data alone. HARMES is freely available at Zenodo, alongside example code for loading the dataset and training models on GitHub.