# An Evolutionary Network Architecture Search Framework with Adaptive Multimodal Fusion for Hand Gesture Recognition

arXiv: https://arxiv.org/abs/2403.18208 · HTML: https://mlsanity.com/paper/2403.18208/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 56 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 7 of 7)

- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Searching for Accurate Binary Neural Architectures](https://mlsanity.com/paper/1909.07378/) (2019)
- [Surface Electromyography as a Natural Human-Machine Interface: A Review](https://mlsanity.com/paper/2101.04658/) (2021)
- [Stable Electromyographic Sequence Prediction During Movement Transitions using Temporal Convolutional Networks](https://mlsanity.com/paper/1901.02442/) (2019)
- [Self-Adaptive 2D-3D Ensemble of Fully Convolutional Networks for Medical Image Segmentation](https://mlsanity.com/paper/1907.11587/) (2019)

## Abstract (arXiv)

Hand gesture recognition (HGR) based on multimodal data has attracted considerable attention owing to its great potential in applications. Various manually designed multimodal deep networks have performed well in multimodal HGR (MHGR), but most of existing algorithms require a lot of expert experience and time-consuming manual trials. To address these issues, we propose an evolutionary network architecture search framework with the adaptive multimodel fusion (AMF-ENAS). Specifically, we design an encoding space that simultaneously considers fusion positions and ratios of the multimodal data, allowing for the automatic construction of multimodal networks with different architectures through decoding. Additionally, we consider three input streams corresponding to intra-modal surface electromyography (sEMG), intra-modal accelerometer (ACC), and inter-modal sEMG-ACC. To automatically adapt to various datasets, the ENAS framework is designed to automatically search a MHGR network with appropriate fusion positions and ratios. To the best of our knowledge, this is the first time that ENAS has been utilized in MHGR to tackle issues related to the fusion position and ratio of multimodal data. Experimental results demonstrate that AMF-ENAS achieves state-of-the-art performance on the Ninapro DB2, DB3, and DB7 datasets.