# Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations

arXiv: https://arxiv.org/abs/2602.08755 · HTML: https://mlsanity.com/paper/2602.08755/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 22 of its 50 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts](https://mlsanity.com/paper/2206.02770/) (2022)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks](https://mlsanity.com/paper/1706.00527/) (2017)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [Multi-level Feature Learning for Contrastive Multi-view Clustering](https://mlsanity.com/paper/2106.11193/) (2021)
- [Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts](https://mlsanity.com/paper/2410.08245/) (2024)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2402.12851/) (2024)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction](https://mlsanity.com/paper/1808.00758/) (2018)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Self-supervised Contrastive Learning of Multi-view Facial Expressions](https://mlsanity.com/paper/2108.06723/) (2021)
- [Virtual Fusion with Contrastive Learning for Single Sensor-based Activity Recognition](https://mlsanity.com/paper/2312.02185/) (2023)
- [Your contrastive learning problem is secretly a distribution alignment problem](https://mlsanity.com/paper/2502.20141/) (2025)
- [SoK: Behind the Accuracy of Complex Human Activity Recognition Using Deep Learning](https://mlsanity.com/paper/2405.00712/) (2024)

## Abstract (arXiv)

Multimodal multiview learning seeks to integrate information from diverse sources to enhance task performance. Existing approaches often struggle with flexible view configurations, including arbitrary view combinations, numbers of views, and heterogeneous modalities. Focusing on the context of human activity recognition, we propose AliAd, a model that combines multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference. Instead of trying to reconstruct missing views, an adjusted center contrastive loss is used for self-supervised representation learning and view alignment, mitigating the impact of missing views on multiview fusion. This loss formulation allows for the integration of view weights to account for view quality. Additionally, it reduces computational complexity from O(V2) to O(V), where V is the number of views. To address residual discrepancies not captured by contrastive learning, we employ a mixture-of-experts module with a specialized load balancing strategy, tasked with adapting to arbitrary view combinations. We highlight the geometric relationship among components in our model and how they combine well in the latent space. AliAd is validated on four datasets encompassing inertial and human pose modalities, with the number of views ranging from three to nine, demonstrating its performance and flexibility.