# Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

arXiv: https://arxiv.org/abs/2602.23694 · HTML: https://mlsanity.com/paper/2602.23694/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [Mixed-Initiative variable autonomy for remotely operated mobile robots](https://mlsanity.com/paper/1911.04848/) (2019)
- [CaptAinGlove: Capacitive and Inertial Fusion-Based Glove for Real-Time on Edge Hand Gesture Recognition for Drone Control](https://mlsanity.com/paper/2306.04319/) (2023)
- [Worker Activity Recognition in Manufacturing Line Using Near-body Electric Field](https://mlsanity.com/paper/2308.03514/) (2023)
- [Cellular-enabled Collaborative Robots Planning and Operations for Search-and-Rescue Scenarios](https://mlsanity.com/paper/2403.09177/) (2024)
- [TeLeMan: Teleoperation for Legged Robot Loco-Manipulation using Wearable IMU-based Motion Capture](https://mlsanity.com/paper/2209.10314/) (2022)
- [Investigating the Usability of Collaborative Robot control through Hands-Free Operation using Eye gaze and Augmented Reality](https://mlsanity.com/paper/2306.13072/) (2023)
- [HORUS: A Mixed Reality Interface for Managing Teams of Mobile Robots](https://mlsanity.com/paper/2506.02622/) (2025)

## Abstract (arXiv)

Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential. In this context, hands-free teleoperation enhances operator mobility and situational awareness, thereby improving safety in hazardous environments. While vision-based gesture recognition has been explored as one method for hands-free teleoperation, its performance often deteriorates under occlusions, lighting variations, and cluttered backgrounds, limiting its applicability in real-world operations. To overcome these limitations, we propose a multimodal gesture recognition framework that integrates inertial data (accelerometer, gyroscope, and orientation) from Apple Watches on both wrists with capacitive sensing signals from custom gloves. We design a late fusion strategy based on the log-likelihood ratio (LLR), which not only enhances recognition performance but also provides interpretability by quantifying modality-specific contributions. To support this research, we introduce a new dataset of 20 distinct gestures inspired by aircraft marshalling signals, comprising synchronized RGB video, IMU, and capacitive sensor data. Experimental results demonstrate that our framework achieves performance comparable to a state-of-the-art vision-based baseline while significantly reducing computational cost, model size, and training time, making it well suited for real-time robot control. We therefore underscore the potential of sensor-based multimodal fusion as a robust and interpretable solution for gesture-driven mobile robot and drone teleoperation.