# Semantic Labeling of Human Action For Visually Impaired And Blind People Scene Interaction

arXiv: https://arxiv.org/abs/2201.04706 · HTML: https://mlsanity.com/paper/2201.04706/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 31 references that are in the map.
- First posted on arXiv 12 Jan 2022.

## Builds on (most cited first, 9 of 9)

- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates](https://mlsanity.com/paper/1706.08276/) (2017)
- [HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition](https://mlsanity.com/paper/1408.3809/) (2014)
- [Smart Guiding Glasses for Visually Impaired People in Indoor Environment](https://mlsanity.com/paper/1709.09359/) (2017)
- [Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences](https://mlsanity.com/paper/1501.04686/) (2015)

## Abstract (arXiv)

The aim of this work is to contribute to the development of a tactile device for visually impaired and blind persons in order to let them to understand actions of the surrounding people and to interact with them. First, based on the state-of-the-art methods of human action recognition from RGB-D sequences, we use the skeleton information provided by Kinect, with the disentangled and unified multi-scale Graph Convolutional (MS-G3D) model to recognize the performed actions. We tested this model on real scenes and found some of constraints and limitations. Next, we apply a fusion between skeleton modality with MS-G3D and depth modality with CNN in order to bypass the discussed limitations. Third, the recognized actions are labeled semantically and will be mapped into an output device perceivable by the touch sense.