M&M: Multimodal-Multitask Model Integrating Audiovisual Cues in Cognitive Load Assessment
Not on the map. Why?
Abstract (arXiv)
This paper introduces the M&M model, a novel multimodal-multitask learning framework, applied to the AVCAffe dataset for cognitive load assessment (CLA). M&M uniquely integrates audiovisual cues through a dual-pathway architecture, featuring specialized streams for audio and video inputs. A key innovation lies in its cross-modality multihead attention mechanism, fusing the different modalities for synchronized multitasking. Another notable feature is the model's three specialized branches, each tailored to a specific cognitive load label, enabling nuanced, task-specific analysis. While it shows modest performance compared to the AVCAffe's single-task baseline, M&M demonstrates a promising framework for integrated multimodal processing. This work paves the way for future enhancements in multimodal-multitask learning systems, emphasizing the fusion of diverse data types for complex task handling.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 10 of its 42 references that are in the map. First posted on arXiv 14 Mar 2024.
Builds on
10 of 42 references.
References · 10
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
- 2022Formal Algorithms for Transformerscited by 70
- 2019SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wildcited by 62
- 2017Identifying beneficial task relations for multi-task learning in deep neural networkscited by 59
- 2022Can CNNs Be More Robust Than Transformers?cited by 38
- 2020Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Learningcited by 8
- 2023MATT: Multimodal Attention Level Estimation for e-learning Platformscited by 7
- 2018Cross-connected Networks for Multi-task Learning of Detection and Segmentationcited by 4
Built on by
Not cited yet.