mlsanityData to 7 Oct 2026

Efficient Gesture Recognition for the Assistance of Visually Impaired People using Multi-Head Neural Networks

arXiv 2205.06980 ↗First posted 14 May 2022cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper proposes an interactive system for mobile devices controlled by hand gestures aimed at helping people with visual impairments. This system allows the user to interact with the device by making simple static and dynamic hand gestures. Each gesture triggers a different action in the system, such as object recognition, scene description or image scaling (e.g., pointing a finger at an object will show a description of it). The system is based on a multi-head neural network architecture, which initially detects and classifies the gestures, and subsequently, depending on the gesture detected, performs a second stage that carries out the corresponding action. This multi-head architecture optimizes the resources required to perform different tasks simultaneously, and takes advantage of the information obtained from an initial backbone to perform different processes in a second stage. To train and evaluate the system, a dataset with about 40k images was manually compiled and labeled including different types of hand gestures, backgrounds (indoors and outdoors), lighting conditions, etc. This dataset contains synthetic gestures (whose objective is to pre-train the system in order to improve the results) and real images captured using different mobile phones. The results obtained and the comparison made with the state of the art show competitive results as regards the different actions performed by the system, such as the accuracy of classification and localization of gestures, or the generation of descriptions for objects and scenes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 84 references that are in the map. First posted on arXiv 14 May 2022.

Builds on

32 of 84 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2018YOLOv3: An Incremental Improvementcited by 2,652
  11. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  12. 2019Searching for MobileNetV3cited by 1,984
20 more
  1. 2014How transferable are features in deep neural networks?cited by 1,216
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  3. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  4. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  5. 2016Image Captioning with Semantic Attentioncited by 260
  6. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  7. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  8. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  9. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
  10. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  11. 2014Return of the Devil in the Details: Delving Deep into Convolutional Netscited by 122
  12. 2019Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networkscited by 30
  13. 2019Fast and Robust Dynamic Hand Gesture Recognition via Key Frames Extraction and Feature Fusioncited by 5
  14. 2020Content and Context Features for Scene Image Representationcited by 5
  15. 2022A Spatio-Temporal Multilayer Perceptron for Gesture Recognitioncited by 5
  16. 2017Where to put the Image in an Image Caption Generatorcited by 4
  17. 2017Using Deep Convolutional Networks for Gesture Recognition in American Sign Languagecited by 4
  18. 2020Scene Image Representation by Foreground, Background and Hybrid Featurescited by 4
  19. 2017Deep Fisher Discriminant Learning for Mobile Hand Gesture Recognitioncited by 3
  20. 2019Hand Segmentation and Fingertip Tracking from Depth Camera Images Using Deep Convolutional Neural Network and Multi-task SegNetcited by 3

Built on by

Not cited yet.