# Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition

arXiv: https://arxiv.org/abs/2505.12007 · HTML: https://mlsanity.com/paper/2505.12007/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 28 of its 54 references that are in the map.
- First posted on arXiv 17 May 2025.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)
- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts](https://mlsanity.com/paper/2206.02770/) (2022)
- [Decoupled Multimodal Distilling for Emotion Recognition](https://mlsanity.com/paper/2303.13802/) (2023)
- [Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2404.04256/) (2024)
- [DEIM: DETR with Improved Matching for Fast Convergence](https://mlsanity.com/paper/2412.04234/) (2024)
- [Hard Mixtures of Experts for Large Scale Weakly Supervised Vision](https://mlsanity.com/paper/1704.06363/) (2017)
- [Fusion-Mamba for Cross-modality Object Detection](https://mlsanity.com/paper/2404.09146/) (2024)
- [CDMamba: Incorporating Local Clues into Mamba for Remote Sensing Image Binary Change Detection](https://mlsanity.com/paper/2406.04207/) (2024)
- [HMoE: Heterogeneous Mixture of Experts for Language Modeling](https://mlsanity.com/paper/2408.10681/) (2024)
- [Yuan 2.0-M32: Mixture of Experts with Attention Router](https://mlsanity.com/paper/2405.17976/) (2024)
- [COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection](https://mlsanity.com/paper/2412.18076/) (2024)
- [Person Re-Identification without Identification via Event Anonymization](https://mlsanity.com/paper/2308.04402/) (2023)
- [DepMamba: Progressive Fusion Mamba for Multimodal Depression Detection](https://mlsanity.com/paper/2409.15936/) (2024)
- [Eyemotion: Classifying facial expressions in VR using eye-tracking cameras](https://mlsanity.com/paper/1707.07204/) (2017)
- [In the Blink of an Eye: Event-based Emotion Recognition](https://mlsanity.com/paper/2310.04043/) (2023)
- [Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognition](https://mlsanity.com/paper/2407.09521/) (2024)
- [I Only Have Eyes for You: The Impact of Masks On Convolutional-Based Facial Expression Recognition](https://mlsanity.com/paper/2104.08353/) (2021)

## Abstract (arXiv)

In this paper, we proposed a Multi-modal Collaborative Optimization and Expansion Network (MCO-E Net), to use event modalities to resist challenges such as low light, high exposure, and high dynamic range in single-eye expression recognition tasks. The MCO-E Net introduces two innovative designs: Multi-modal Collaborative Optimization Mamba (MCO-Mamba) and Heterogeneous Collaborative and Expansion Mixture-of-Experts (HCE-MoE). MCO-Mamba, building upon Mamba, leverages dual-modal information to jointly optimize the model, facilitating collaborative interaction and fusion of modal semantics. This approach encourages the model to balance the learning of both modalities and harness their respective strengths. HCE-MoE, on the other hand, employs a dynamic routing mechanism to distribute structurally varied experts (deep, attention, and focal), fostering collaborative learning of complementary semantics. This heterogeneous architecture systematically integrates diverse feature extraction paradigms to comprehensively capture expression semantics. Extensive experiments demonstrate that our proposed network achieves competitive performance in the task of single-eye expression recognition, especially under poor lighting conditions.