# Efficient-Husformer: Efficient Multimodal Transformer Hyperparameter Optimization for Stress and Cognitive Loads

arXiv: https://arxiv.org/abs/2511.22362 · HTML: https://mlsanity.com/paper/2511.22362/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Low-Rank Bottleneck in Multi-head Attention Models](https://mlsanity.com/paper/2002.07028/) (2020)
- [A Close Look at Deep Learning with Small Data](https://mlsanity.com/paper/2003.12843/) (2020)
- [On the Transformer Growth for Progressive BERT Training](https://mlsanity.com/paper/2010.12562/) (2020)
- [Staged Training for Transformer Language Models](https://mlsanity.com/paper/2203.06211/) (2022)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [A Transformer Architecture for Stress Detection from ECG](https://mlsanity.com/paper/2108.09737/) (2021)
- [DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework](https://mlsanity.com/paper/1706.01215/) (2017)
- [Convolutional RNN: an Enhanced Model for Extracting Features from Sequential Data](https://mlsanity.com/paper/1602.05875/) (2016)
- [Transformer-Based Self-Supervised Learning for Emotion Recognition](https://mlsanity.com/paper/2204.05103/) (2022)
- [One Wide Feedforward is All You Need](https://mlsanity.com/paper/2309.01826/) (2023)
- [Minusformer: Improving Time Series Forecasting by Progressively Learning Residuals](https://mlsanity.com/paper/2402.02332/) (2024)
- [Hyperparameter Importance Analysis for Multi-Objective AutoML](https://mlsanity.com/paper/2405.07640/) (2024)
- [Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensors](https://mlsanity.com/paper/2005.03948/) (2020)
- [Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition](https://mlsanity.com/paper/2209.15182/) (2022)
- [Efficient Training of Large Vision Models via Advanced Automated Progressive Learning](https://mlsanity.com/paper/2410.00350/) (2024)
- [MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Task](https://mlsanity.com/paper/2105.08146/) (2021)
- [Number of Attention Heads vs Number of Transformer-Encoders in Computer Vision](https://mlsanity.com/paper/2209.07221/) (2022)
- [Multi-scale Transformer-based Network for Emotion Recognition from Multi Physiological Signals](https://mlsanity.com/paper/2305.00769/) (2023)

## Abstract (arXiv)

Transformer-based models have gained considerable attention in the field of physiological signal analysis. They leverage long-range dependencies and complex patterns in temporal signals, allowing them to achieve performance superior to traditional RNN and CNN models. However, they require high computational intensity and memory demands. In this work, we present Efficient-Husformer, a novel Transformer-based architecture developed with hyperparameter optimization (HPO) for multi-class stress detection across two multimodal physiological datasets (WESAD and CogLoad). The main contributions of this work are: (1) the design of a structured search space, targeting effective hyperparameter optimization; (2) a comprehensive ablation study evaluating the impact of architectural decisions; (3) consistent performance improvements over the original Husformer, with the best configuration achieving an accuracy of 88.41 and 92.61 (improvements of 13.83% and 6.98%) on WESAD and CogLoad datasets, respectively. The best-performing configuration is achieved with the (L + dm) or (L + FFN) modality combinations, using a single layer, 3 attention heads, a model dimension of 18/30, and FFN dimension of 120/30, resulting in a compact model with only about 30k parameters.