Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations
On the map: Mixture-of-experts → Multimodal MoE models
Abstract (arXiv)
Multimodal multiview learning seeks to integrate information from diverse sources to enhance task performance. Existing approaches often struggle with flexible view configurations, including arbitrary view combinations, numbers of views, and heterogeneous modalities. Focusing on the context of human activity recognition, we propose AliAd, a model that combines multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference. Instead of trying to reconstruct missing views, an adjusted center contrastive loss is used for self-supervised representation learning and view alignment, mitigating the impact of missing views on multiview fusion. This loss formulation allows for the integration of view weights to account for view quality. Additionally, it reduces computational complexity from O(V2) to O(V), where V is the number of views. To address residual discrepancies not captured by contrastive learning, we employ a mixture-of-experts module with a specialized load balancing strategy, tasked with adapting to arbitrary view combinations. We highlight the geometric relationship among components in our model and how they combine well in the latent space. AliAd is validated on four datasets encompassing inertial and human pose modalities, with the number of views ranging from three to nine, demonstrating its performance and flexibility.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 22 of its 50 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
22 of 50 references.
Closest ancestors · 2
Same direction, newest first
- 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts10 Oct
- 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun
Other references · 20
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
- 2019Contrastive Multiview Codingcited by 1,297
- 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
- 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
- 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 159
- 2017Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networkscited by 130
- 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
- 2021Multi-level Feature Learning for Contrastive Multi-view Clusteringcited by 73
- 2024Multimodal Alignment and Fusion: A Surveycited by 73
- 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
8 more
- 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Modelscited by 54
- 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 53
- 2018Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstructioncited by 44
- 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
- 2021Self-supervised Contrastive Learning of Multi-view Facial Expressionscited by 23
- 2023Virtual Fusion with Contrastive Learning for Single Sensor-based Activity Recognitioncited by 9
- 2025Your contrastive learning problem is secretly a distribution alignment problemcited by 6
- 2024SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SkillMoV: Mixture-of-View Routing with Prototype-Conditioned Gating for Unified Multi-View Proficiency Estimation
- 2023Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception
- 2026SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity
- 2026Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
- 2025Few-shot Vision-based Human Activity Recognition with MLLM-based Visual Reinforcement Learning
- 2026Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
- 2023An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training
- 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations