# Robotic Detection of a Human-Comprehensible Gestural Language for Underwater Multi-Human-Robot Collaboration

arXiv: https://arxiv.org/abs/2207.05331 · HTML: https://mlsanity.com/paper/2207.05331/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Fast Underwater Image Enhancement for Improved Visual Perception](https://mlsanity.com/paper/1903.09766/) (2019)
- [ActionVLAD: Learning spatio-temporal aggregation for action classification](https://mlsanity.com/paper/1704.02895/) (2017)
- [VideoLSTM Convolves, Attends and Flows for Action Recognition](https://mlsanity.com/paper/1607.01794/) (2016)
- [Towards Good Practices for Very Deep Two-Stream ConvNets](https://mlsanity.com/paper/1507.02159/) (2015)
- [Robotic Detection of Marine Litter Using Deep Visual Detection Models](https://mlsanity.com/paper/1804.01079/) (2018)
- [Navigation in the Presence of Obstacles for an Agile Autonomous Underwater Vehicle](https://mlsanity.com/paper/1903.11750/) (2019)
- [Underwater Image Super-Resolution using Deep Residual Multipliers](https://mlsanity.com/paper/1909.09437/) (2019)
- [Understanding Human Motion and Gestures for Underwater Human-Robot Collaboration](https://mlsanity.com/paper/1804.02479/) (2018)
- [Design and Experiments with LoCO AUV: A Low Cost Open-Source Autonomous Underwater Vehicle](https://mlsanity.com/paper/2003.09041/) (2020)
- [One-Shot Informed Robotic Visual Search in the Wild](https://mlsanity.com/paper/2003.10010/) (2020)
- [Dynamic Reconfiguration of Mission Parameters in Underwater Human-Robot Collaboration](https://mlsanity.com/paper/1709.08772/) (2017)
- [Robot Communication Via Motion: Closing the Underwater Human-Robot Interaction Loop](https://mlsanity.com/paper/1809.07948/) (2018)
- [Visual Diver Face Recognition for Underwater Human-Robot Interaction](https://mlsanity.com/paper/2011.09556/) (2020)

## Built on by (most cited first, 2 of 2)

- [RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations](https://mlsanity.com/paper/2502.13134/) (2025)
- [Semantically-Aware Diver Activity Recognition Framework for Effective Underwater Multi-Human-Robot Collaboration](https://mlsanity.com/paper/2606.12374/) (2026)

## Abstract (arXiv)

In this paper, we present a motion-based robotic communication framework that enables non-verbal communication among autonomous underwater vehicles (AUVs) and human divers. We design a gestural language for AUV-to-AUV communication which can be easily understood by divers observing the conversation unlike typical radio frequency, light, or audio based AUV communication. To allow AUVs to visually understand a gesture from another AUV, we propose a deep network (RRCommNet) which exploits a self-attention mechanism to learn to recognize each message by extracting maximally discriminative spatio-temporal features. We train this network on diverse simulated and real-world data. Our experimental evaluations, both in simulation and in closed-water robot trials, demonstrate that the proposed RRCommNet architecture is able to decipher gesture-based messages with an average accuracy of 88-94% on simulated data, 73-83% on real data (depending on the version of the model used). Further, by performing a message transcription study with human participants, we also show that the proposed language can be understood by humans, with an overall transcription accuracy of 88%. Finally, we discuss the inference runtime of RRCommNet on embedded GPU hardware, for real-time use on board AUVs in the field.