# MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

arXiv: https://arxiv.org/abs/2302.14595 · HTML: https://mlsanity.com/paper/2302.14595/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World](https://mlsanity.com/paper/2107.03172/) (2021)
- [A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing](https://mlsanity.com/paper/2207.10614/) (2022)
- [Trans4Trans: Efficient Transformer for Transparent Object and Semantic Scene Segmentation in Real-World Navigation Assistance](https://mlsanity.com/paper/2108.09174/) (2021)
- [SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognition](https://mlsanity.com/paper/2301.13156/) (2023)
- [HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor](https://mlsanity.com/paper/2107.03180/) (2021)
- [Panoptic Lintention Network: Towards Efficient Navigational Perception for the Visually Impaired](https://mlsanity.com/paper/2103.04128/) (2021)
- [Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor](https://mlsanity.com/paper/2204.01154/) (2022)

## Built on by (most cited first, 9 of 9)

- [Open Panoramic Segmentation](https://mlsanity.com/paper/2407.02685/) (2024)
- [Referring Atomic Video Action Recognition](https://mlsanity.com/paper/2407.01872/) (2024)
- [Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments](https://mlsanity.com/paper/2307.07757/) (2023)
- [MICA: Multi-Agent Industrial Coordination Assistant](https://mlsanity.com/paper/2509.15237/) (2025)
- [Deformable Mamba for Wide Field of View Segmentation](https://mlsanity.com/paper/2411.16481/) (2024)
- [UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields](https://mlsanity.com/paper/2506.21884/) (2025)
- [Learning Zero-Shot Material States Segmentation, by Implanting Natural Image Patterns in Synthetic Data](https://mlsanity.com/paper/2403.03309/) (2024)
- [@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology](https://mlsanity.com/paper/2409.14215/) (2024)
- [Position: Assistive Agents Need Accessibility Alignment](https://mlsanity.com/paper/2605.13579/) (2026)

## Abstract (arXiv)

People with Visual Impairments (PVI) typically recognize objects through haptic perception. Knowing objects and materials before touching is desired by the target users but under-explored in the field of human-centered robotics. To fill this gap, in this work, a wearable vision-based robotic system, MateRobot, is established for PVI to recognize materials and object categories beforehand. To address the computational constraints of mobile platforms, we propose a lightweight yet accurate model MateViT to perform pixel-wise semantic segmentation, simultaneously recognizing both objects and materials. Our methods achieve respective 40.2% and 51.1% of mIoU on COCOStuff-10K and DMS datasets, surpassing the previous method with +5.7% and +7.0% gains. Moreover, on the field test with participants, our wearable system reaches a score of 28 in the NASA-Task Load Index, indicating low cognitive demands and ease of use. Our MateRobot demonstrates the feasibility of recognizing material property through visual cues and offers a promising step towards improving the functionality of wearable robots for PVI. The source code has been made publicly available at https://junweizheng93.github.io/publications/MATERobot/MATERobot.html.