# FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations

arXiv: https://arxiv.org/abs/2602.17573 · HTML: https://mlsanity.com/paper/2602.17573/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 61 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Vision GNN: An Image is Worth Graph of Nodes](https://mlsanity.com/paper/2206.00272/) (2022)
- [FLatten Transformer: Vision Transformer using Focused Linear Attention](https://mlsanity.com/paper/2308.00442/) (2023)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Two-Stream Network for Sign Language Recognition and Translation](https://mlsanity.com/paper/2211.01367/) (2022)
- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding](https://mlsanity.com/paper/2305.04868/) (2023)
- [HaGRID - HAnd Gesture Recognition Image Dataset](https://mlsanity.com/paper/2206.08219/) (2022)
- [BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization](https://mlsanity.com/paper/2302.05075/) (2023)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [UAV-GESTURE: A Dataset for UAV Control and Gesture Recognition](https://mlsanity.com/paper/1901.02602/) (2019)
- [Self-supervised visual learning in the low-data regime: a comparative evaluation](https://mlsanity.com/paper/2404.17202/) (2024)
- [Survey on Hand Gesture Recognition from Visual Input](https://mlsanity.com/paper/2501.11992/) (2025)
- [Towards open and expandable cognitive AI architectures for large-scale multi-agent human-robot collaborative learning](https://mlsanity.com/paper/2012.08174/) (2020)
- [User profile-driven large-scale multi-agent learning from demonstration in federated human-robot collaborative environments](https://mlsanity.com/paper/2103.16434/) (2021)
- [ParameterNet: Parameters Are All You Need](https://mlsanity.com/paper/2306.14525/) (2023)
- [Ultra-Range Gesture Recognition using a Web-Camera in Human-Robot Interaction](https://mlsanity.com/paper/2311.15361/) (2023)
- [Exploring Energy Landscapes for Minimal Counterfactual Explanations: Applications in Cybersecurity and Beyond](https://mlsanity.com/paper/2503.18185/) (2025)
- [Distributed maze exploration using multiple agents and optimal goal assignment](https://mlsanity.com/paper/2405.20232/) (2024)

## Built on by (most cited first, 1 of 1)

- [Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision](https://mlsanity.com/paper/2603.26646/) (2026)

## Abstract (arXiv)

The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: https://doi.org/10.5281/zenodo.18131333.