# Perception Framework through Real-Time Semantic Segmentation and Scene Recognition on a Wearable System for the Visually Impaired

arXiv: https://arxiv.org/abs/2103.04136 · HTML: https://mlsanity.com/paper/2103.04136/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 6 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [LambdaNetworks: Modeling Long-Range Interactions Without Attention](https://mlsanity.com/paper/2102.08602/) (2021)
- [In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images](https://mlsanity.com/paper/1903.08469/) (2019)
- [Semantic-Aware Scene Recognition](https://mlsanity.com/paper/1909.02410/) (2019)
- [Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images](https://mlsanity.com/paper/2002.10570/) (2020)
- [Real-time Semantic Segmentation with Fast Attention](https://mlsanity.com/paper/2007.03815/) (2020)
- [Light-Weight RefineNet for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1810.03272/) (2018)
- [Deep Learning based Wearable Assistive System for Visually Impaired People](https://mlsanity.com/paper/1908.03364/) (2019)
- [Weakly Supervised PatchNets: Describing and Aggregating Local Patches for Scene Recognition](https://mlsanity.com/paper/1609.00153/) (2016)
- [In pixels we trust: From Pixel Labeling to Object Localization and Scene Categorization](https://mlsanity.com/paper/1807.07284/) (2018)
- [Can we cover navigational perception needs of the visually impaired by panoptic segmentation?](https://mlsanity.com/paper/2007.10202/) (2020)

## Built on by (most cited first, 3 of 3)

- [HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor](https://mlsanity.com/paper/2107.03180/) (2021)
- [ViT Cane: Visual Assistant for the Visually Impaired](https://mlsanity.com/paper/2109.13857/) (2021)
- [A Deep Learning-based Global and Segmentation-based Semantic Feature Fusion Approach for Indoor Scene Classification](https://mlsanity.com/paper/2302.06432/) (2023)

## Abstract (arXiv)

As the scene information, including objectness and scene type, are important for people with visual impairment, in this work we present a multi-task efficient perception system for the scene parsing and recognition tasks. Building on the compact ResNet backbone, our designed network architecture has two paths with shared parameters. In the structure, the semantic segmentation path integrates fast attention, with the aim of harvesting long-range contextual information in an efficient manner. Simultaneously, the scene recognition path attains the scene type inference by passing the semantic features into semantic-driven attention networks and combining the semantic extracted representations with the RGB extracted representations through a gated attention module. In the experiments, we have verified the systems' accuracy and efficiency on both public datasets and real-world scenes. This system runs on a wearable belt with an Intel RealSense LiDAR camera and an Nvidia Jetson AGX Xavier processor, which can accompany visually impaired people and provide assistive scene information in their navigation tasks.