# Towards a Unified Modality-Agnostic Multimodal Framework for Cognitive Workload Assessment

arXiv: https://arxiv.org/abs/2609.20199 · HTML: https://mlsanity.com/paper/2609.20199/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 25 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Efficient Pain Recognition via Respiration Signals: A Single Cross-Attention Transformer Multi-Window Fusion Pipeline](https://mlsanity.com/paper/2507.21886/) (2025)
- [PainFormer: a Vision Foundation Model for Automatic Pain Assessment](https://mlsanity.com/paper/2505.01571/) (2025)
- [Multi-Representation Diagrams for Pain Recognition: Integrating Various Electrodermal Activity Signals into a Single Image](https://mlsanity.com/paper/2507.21881/) (2025)
- [Tiny-BioMoE: a Lightweight Embedding Model for Biosignal Analysis](https://mlsanity.com/paper/2507.21875/) (2025)
- [A Lightweight Transformer for Pain Recognition from Brain Activity](https://mlsanity.com/paper/2604.16491/) (2026)
- [A Unified Tokenization Framework for Pain Recognition using Heterogeneous 3D Modalities](https://mlsanity.com/paper/2607.19716/) (2026)
- [ReFace: Reorganizing Facial Spatiotemporal Representations for Improved Pain Assessment](https://mlsanity.com/paper/2607.19722/) (2026)
- [Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech](https://mlsanity.com/paper/2604.11417/) (2026)
- [Measuring Cognitive Workload Using Multimodal Sensors](https://mlsanity.com/paper/2205.04235/) (2022)
- [Neural Networks Meet Neural Activity: Utilizing EEG for Mental Workload Estimation](https://mlsanity.com/paper/2408.13930/) (2024)
- [One-Block Transformer (1BT) for EEG-Based Cognitive Workload Assessment](https://mlsanity.com/paper/2605.00856/) (2026)

## Built on by (most cited first, 5 of 5)

- [Beyond the Raw Waveform: Fusing Visual Representations of EDA for Stress Detection](https://mlsanity.com/paper/2609.22095/) (2026)
- [A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition](https://mlsanity.com/paper/2608.09088/) (2026)
- [FUSE: Frame-Unified Stress Estimation from Facial Video](https://mlsanity.com/paper/2608.10442/) (2026)
- [MUPA2E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment](https://mlsanity.com/paper/2608.15999/) (2026)
- [UNWIND: Any-Length Facial Video for Stress Detection without Temporal Windowing](https://mlsanity.com/paper/2609.29553/) (2026)

## Abstract (arXiv)

Cognitive workload reflects the mental effort required during task performance and is central to the design of adaptive human-machine systems. The use of biosignals to measure cognitive workload has been extensively researched and documented; however, studies examining the effects of combining heterogeneous biosignal modalities for this purpose remain limited. To provide insight into this area, we developed a unified, modality-agnostic, hierarchical Transformer-based architecture to process heterogeneous biosignal modalities within a single model. We use this framework in a pilot study evaluating all 31 possible combinations of five modalities: Electrocardiogram (ECG), Electrodermal Activity (EDA), Respiration (RESP), Peripheral Oxygen Saturation (SpO2), and Electroencephalogram (EEG), under leave-one-subject-out validation across three cognitively distinct tasks: abstract reasoning (IQ), arithmetic problem solving (MATH), and a game task (GAME). In this pilot setting, the results suggest that: (i) EEG is the strongest single modality, ranking highest in IQ, GAME, and the pooled ALL setting, where samples from all three tasks are combined; (ii) adding more modalities does not consistently improve performance; (iii) the full five-modality combination achieves the highest textitAverage score of 73.02% on IQ and 68.08% when the textitAverage scores are averaged over the four evaluation settings: IQ, MATH, GAME, and ALL; and (iv) the proposed method reduces model size by approximately 50% compared with late-fusion alternatives while maintaining a lower inference time.