# NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment

arXiv: https://arxiv.org/abs/2509.18672 · HTML: https://mlsanity.com/paper/2509.18672/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 69 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Builds on (most cited first, 11 of 11)

- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments](https://mlsanity.com/paper/2404.02508/) (2024)
- [Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation System](https://mlsanity.com/paper/2410.04005/) (2024)
- [Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence](https://mlsanity.com/paper/2106.08710/) (2021)
- [WanderGuide: Indoor Map-less Robotic Guide for Exploration by Blind People](https://mlsanity.com/paper/2502.08906/) (2025)
- [ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Programming](https://mlsanity.com/paper/2408.10499/) (2024)
- [ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People](https://mlsanity.com/paper/2412.03118/) (2024)

## Built on by (most cited first, 2 of 2)

- [VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility](https://mlsanity.com/paper/2607.16750/) (2026)
- [Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments](https://mlsanity.com/paper/2609.21828/) (2026)

## Abstract (arXiv)

People with visual impairments often face significant challenges in locating and retrieving objects in their surroundings. Existing assistive technologies present a trade-off: systems that offer precise guidance typically require pre-scanning or support only fixed object categories, while those with open-world object recognition lack spatial feedback for reaching the object. To address this gap, we introduce 'NaviSense', a mobile assistive system that combines conversational AI, vision-language models, augmented reality (AR), and LiDAR to support open-world object detection with real-time audio-haptic guidance. Users specify objects via natural language and receive continuous spatial feedback to navigate toward the target without needing prior setup. Designed with insights from a formative study and evaluated with 12 blind and low-vision participants, NaviSense significantly reduced object retrieval time and was preferred over existing tools, demonstrating the value of integrating open-world perception with precise, accessible guidance.