# Can ChatGPT assist visually impaired people with micro-navigation?

arXiv: https://arxiv.org/abs/2408.08321 · HTML: https://mlsanity.com/paper/2408.08321/

## Key facts

- Cited by 2 papers in the map (data through 7 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [SOON: Scenario Oriented Object Navigation with Graph-based Exploration](https://mlsanity.com/paper/2103.17138/) (2021)
- [L3MVN: Leveraging Large Language Models for Visual Target Navigation](https://mlsanity.com/paper/2304.05501/) (2023)
- [SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.14143/) (2021)
- [Learning To Follow Directions in Street View](https://mlsanity.com/paper/1903.00401/) (2019)
- [Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration](https://mlsanity.com/paper/2203.04006/) (2022)
- [Multimodal Large Language Model for Visual Navigation](https://mlsanity.com/paper/2310.08669/) (2023)
- [Field evaluation of a mobile app for assisting blind and visually impaired travelers to find bus stops](https://mlsanity.com/paper/2309.10940/) (2023)

## Built on by (most cited first, 2 of 2)

- [Position: Assistive Agents Need Accessibility Alignment](https://mlsanity.com/paper/2605.13579/) (2026)
- [MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places](https://mlsanity.com/paper/2608.28384/) (2026)

## Abstract (arXiv)

Objective: Micro-navigation poses challenges for blind and visually impaired individuals. They often need to ask for sighted assistance. We explored the feasibility of utilizing ChatGPT as a virtual assistant to provide navigation directions. Methods: We created a test set of outdoor and indoor micro-navigation scenarios consisting of 113 scene images and their human-generated text descriptions. A total of 412 way-finding queries and their expected responses were compiled based on the scenarios. Not all queries are answerable based on the information available in the scene image. "I do not know"response was expected for unanswerable queries, which served as negative cases. High level orientation responses were expected, and step-by-step guidance was not required. ChatGPT 4o was evaluated based on sensitivity (SEN) and specificity (SPE) under different conditions. Results: The default ChatGPT 4o, with scene images as inputs, resulted in SEN and SPE values of 64.8% and 75.9%, respectively. Instruction on how to respond to unanswerable questions did not improve SEN substantially but SPE increased by around 14 percentage points. SEN and SPE both improved substantially, by about 17 and 16 percentage points on average respectively, when human written descriptions of the scenes were provided as input instead of images. Providing further prompt instructions to the assistants when the input was text description did not substantially change the SEN and SPE values. Conclusion: Current native ChatGPT 4o is still unable to provide correct micro-navigation guidance in some cases, probably because its scene understanding is not optimized for navigation purposes. If multi-modal chatbots could interpret scenes with a level of clarity comparable to humans, and also guided by appropriate prompts, they may have the potential to provide assistance to visually impaired for micro-navigation.