# IsoNet: Causal Analysis of Multimodal Transformers for Neuromuscular Gesture Classification

arXiv: https://arxiv.org/abs/2506.16744 · HTML: https://mlsanity.com/paper/2506.16744/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP](https://mlsanity.com/paper/2308.14179/) (2023)
- [Is Cross-Attention Preferable to Self-Attention for Multi-Modal Emotion Recognition?](https://mlsanity.com/paper/2202.09263/) (2022)
- [Force-Aware Interface via Electromyography for Natural VR/AR Interaction](https://mlsanity.com/paper/2210.01225/) (2022)
- [Force-guided High-precision Grasping Control of Fragile and Deformable Objects using sEMG-based Force Prediction](https://mlsanity.com/paper/2002.01791/) (2020)

## Abstract (arXiv)

Hand gestures are a primary output of the human motor system, yet the decoding of their neuromuscular signatures remains a bottleneck for basic neuroscience and assistive technologies such as prosthetics. Traditional human-machine interface pipelines rely on a single biosignal modality, but multimodal fusion can exploit complementary information from sensors. We systematically compare linear and attention-based fusion strategies across three architectures: a Multimodal MLP, a Multimodal Transformer, and a Hierarchical Transformer, evaluating performance on scenarios with unimodal and multimodal inputs. Experiments use two publicly available datasets: NinaPro DB2 (sEMG and accelerometer) and HD-sEMG 65-Gesture (high-density sEMG and force). Across both datasets, the Hierarchical Transformer with attention-based fusion consistently achieved the highest accuracy, surpassing the multimodal and best single-modality linear-fusion MLP baseline by over 10% on NinaPro DB2 and 3.7% on HD-sEMG. To investigate how modalities interact, we introduce an Isolation Network that selectively silences unimodal or cross-modal attention pathways, quantifying each group of token interactions' contribution to downstream decisions. Ablations reveal that cross-modal interactions contribute approximately 30% of the decision signal across transformer layers, highlighting the importance of attention-driven fusion in harnessing complementary modality information. Together, these findings reveal when and how multimodal fusion would enhance biosignal classification and also provides mechanistic insights of human muscle activities. The study would be beneficial in the design of sensor arrays for neurorobotic systems.