# Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation System

arXiv: https://arxiv.org/abs/2410.04005 · HTML: https://mlsanity.com/paper/2410.04005/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2024.

## Built on by (most cited first, 13 of 13)

- [Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models](https://mlsanity.com/paper/2502.16098/) (2025)
- [ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People](https://mlsanity.com/paper/2412.03118/) (2024)
- [AccessEval: Benchmarking Disability Bias in Large Language Models](https://mlsanity.com/paper/2509.22703/) (2025)
- [LLM-Guided Indoor Navigation with Multimodal Map Understanding](https://mlsanity.com/paper/2503.11702/) (2025)
- [Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impaired](https://mlsanity.com/paper/2508.03651/) (2025)
- [NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment](https://mlsanity.com/paper/2509.18672/) (2025)
- [Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users](https://mlsanity.com/paper/2602.16930/) (2026)
- [IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth](https://mlsanity.com/paper/2505.22305/) (2025)
- [Evaluating OCR Performance for Assistive Technology: Effects of Walking Speed, Camera Placement, and Camera Type](https://mlsanity.com/paper/2602.02223/) (2026)
- [Position: Assistive Agents Need Accessibility Alignment](https://mlsanity.com/paper/2605.13579/) (2026)
- [VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility](https://mlsanity.com/paper/2607.16750/) (2026)
- [Time-Aware Assistive Navigation](https://mlsanity.com/paper/2609.05596/) (2026)
- [Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments](https://mlsanity.com/paper/2609.21828/) (2026)

## Abstract (arXiv)

Assistive technologies for people with visual impairments (PVI) have made significant advancements, particularly with the integration of artificial intelligence (AI) and real-time sensor technologies. However, current solutions often require PVI to switch between multiple apps and tools for tasks like image recognition, navigation, and obstacle detection, which can hinder a seamless and efficient user experience. In this paper, we present NaviGPT, a high-fidelity prototype that integrates LiDAR-based obstacle detection, vibration feedback, and large language model (LLM) responses to provide a comprehensive and real-time navigation aid for PVI. Unlike existing applications such as Be My AI and Seeing AI, NaviGPT combines image recognition and contextual navigation guidance into a single system, offering continuous feedback on the user's surroundings without the need for app-switching. Meanwhile, NaviGPT compensates for the response delays of LLM by using location and sensor data, aiming to provide practical and efficient navigation support for PVI in dynamic environments.