# HAR-DoReMi: Optimizing Data Mixture for Self-Supervised Human Activity Recognition Across Heterogeneous IMU Datasets

arXiv: https://arxiv.org/abs/2503.13542 · HTML: https://mlsanity.com/paper/2503.13542/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 63 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [SelfReg: Self-supervised Contrastive Regularization for Domain Generalization](https://mlsanity.com/paper/2104.09841/) (2021)
- [Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks](https://mlsanity.com/paper/1706.00527/) (2017)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing](https://mlsanity.com/paper/1611.01942/) (2016)
- [Mobile Sensor Data Anonymization](https://mlsanity.com/paper/1810.11546/) (2018)
- [Deep Learning for Time-Series Analysis](https://mlsanity.com/paper/1701.01887/) (2017)
- [What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?](https://mlsanity.com/paper/2202.05998/) (2022)
- [Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification](https://mlsanity.com/paper/2208.06616/) (2022)
- [Minimax Regret Optimization for Robust Machine Learning under Distribution Shift](https://mlsanity.com/paper/2202.05436/) (2022)
- [Out-of-Distribution Representation Learning for Time Series Classification](https://mlsanity.com/paper/2209.07027/) (2022)
- [Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition](https://mlsanity.com/paper/2206.06629/) (2022)
- [UniMTS: Unified Pre-training for Motion Time Series](https://mlsanity.com/paper/2410.19818/) (2024)
- [SWL-Adapt: An Unsupervised Domain Adaptation Model with Sample Weight Learning for Cross-User Wearable Human Activity Recognition](https://mlsanity.com/paper/2212.00724/) (2022)
- [Domain Generalization for Activity Recognition via Adaptive Feature Fusion](https://mlsanity.com/paper/2207.11221/) (2022)
- [Domain-Specific Risk Minimization for Out-of-Distribution Generalization](https://mlsanity.com/paper/2208.08661/) (2022)
- [Transfer Learning in Human Activity Recognition: A Survey](https://mlsanity.com/paper/2401.10185/) (2024)
- [Incremental Real-Time Personalization in Human Activity Recognition Using Domain Adaptive Batch Normalization](https://mlsanity.com/paper/2005.12178/) (2020)
- [Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning](https://mlsanity.com/paper/2306.04641/) (2023)

## Built on by (most cited first, 2 of 2)

- [Towards Generalizable Human Activity Recognition: A Survey](https://mlsanity.com/paper/2508.12213/) (2025)
- [Channel-Free Human Activity Recognition via Inductive-Bias-Aware Fusion Design for Heterogeneous IoT Sensor Environments](https://mlsanity.com/paper/2604.21369/) (2026)

## Abstract (arXiv)

Cross-dataset Human Activity Recognition (HAR) suffers from limited model generalization, hindering its practical deployment. To address this critical challenge, inspired by the success of DoReMi in Large Language Models (LLMs), we introduce a data mixture optimization strategy for pre-training HAR models, aiming to improve the recognition performance across heterogeneous datasets. However, directly applying DoReMi to the HAR field encounters new challenges due to the continuous, multi-channel and intrinsic heterogeneous characteristics of IMU sensor data. To overcome these limitations, we propose a novel framework HAR-DoReMi, which introduces a masked reconstruction task based on Mean Squared Error (MSE) loss. By raplacing the discrete language sequence prediction task, which relies on the Negative Log-Likelihood (NLL) loss, in the original DoReMi framework, the proposed framework is inherently more appropriate for handling the continuous and multi-channel characteristics of IMU data. In addition, HAR-DoReMi integrates the Mahony fusion algorithm into the self-supervised HAR pre-training, aiming to mitigate the heterogeneity of varying sensor orientation. This is achieved by estimating the sensor orientation within each dataset and facilitating alignment with a unified coordinate system, thereby improving the cross-dataset generalization ability of the HAR model. Experimental evaluation on multiple cross-dataset HAR transfer tasks demonstrates that HAR-DoReMi improves the accuracy by an average of 6.51%, compared to the current state-of-the-art method with only approximately 30% to 50% of the data usage. These results confirm the effectiveness of HAR-DoReMi in improving the generalization and data efficiency of pre-training HAR models, underscoring its significant potential to facilitate the practical deployment of HAR technology.