# ShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic Cane

arXiv: https://arxiv.org/abs/2405.20501 · HTML: https://mlsanity.com/paper/2405.20501/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 52 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 4 of 4)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [Precise Detection in Densely Packed Scenes](https://mlsanity.com/paper/1904.00853/) (2019)
- [Robotic Guide Dog: Leading a Human with Leash-Guided Hybrid Physical Interaction](https://mlsanity.com/paper/2103.14300/) (2021)

## Built on by (most cited first, 3 of 3)

- [ShelfAware: Real-Time Semantic Localization in Quasi-Static Environments with Low-Cost Sensors](https://mlsanity.com/paper/2512.09065/) (2025)
- [GIST: Multimodal Knowledge Extraction and Spatial Grounding via Intelligent Semantic Topology](https://mlsanity.com/paper/2604.15495/) (2026)
- [RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Vision](https://mlsanity.com/paper/2609.20330/) (2026)

## Abstract (arXiv)

The ability to shop independently, especially in grocery stores, is important for maintaining a high quality of life. This can be particularly challenging for people with visual impairments (PVI). Stores carry thousands of products, with approximately 30,000 new products introduced each year in the US market alone, presenting a challenge even for modern computer vision solutions. Through this work, we present a proof-of-concept socially assistive robotic system we call ShelfHelp, and propose novel technical solutions for enhancing instrumented canes traditionally meant for navigation tasks with additional capability within the domain of shopping. ShelfHelp includes a novel visual product locator algorithm designed for use in grocery stores and a novel planner that autonomously issues verbal manipulation guidance commands to guide the user during product retrieval. Through a human subjects study, we show the system's success in locating and providing effective manipulation guidance to retrieve desired products with novice users. We compare two autonomous verbal guidance modes achieving comparable performance to a human assistance baseline and present encouraging findings that validate our system's efficiency and effectiveness and through positive subjective metrics including competence, intelligence, and ease of use.