# StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AI

arXiv: https://arxiv.org/abs/2508.08524 · HTML: https://mlsanity.com/paper/2508.08524/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 130 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 26 of 26)

- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model](https://mlsanity.com/paper/2403.05034/) (2024)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [GIS Copilot: Towards an Autonomous GIS Agent for Spatial Analysis](https://mlsanity.com/paper/2411.03205/) (2024)
- [Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People](https://mlsanity.com/paper/2403.15604/) (2024)
- [Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge](https://mlsanity.com/paper/2012.11696/) (2020)
- [Fakes of Varying Shades: How Warning Affects Human Perception and Engagement Regarding LLM Hallucinations](https://mlsanity.com/paper/2404.03745/) (2024)
- [CitySurfaces: City-Scale Semantic Segmentation of Sidewalk Materials](https://mlsanity.com/paper/2201.02260/) (2022)
- [Context-Aware Image Descriptions for Web Accessibility](https://mlsanity.com/paper/2409.03054/) (2024)
- [SonifyAR: Context-Aware Sound Generation in Augmented Reality](https://mlsanity.com/paper/2405.07089/) (2024)
- [CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision](https://mlsanity.com/paper/2407.13515/) (2024)
- [Towards Global-Scale Crowd+AI Techniques to Map and Assess Sidewalks for People with Disabilities](https://mlsanity.com/paper/2206.13677/) (2022)
- [Engaging with Children's Artwork in Mixed Visual-Ability Families](https://mlsanity.com/paper/2407.18874/) (2024)
- [When Refreshable Tactile Displays Meet Conversational Agents: Investigating Accessible Data Presentation and Analysis with Touch and Speech](https://mlsanity.com/paper/2408.04806/) (2024)
- [An AI Guide to Enhance Accessibility of Social Virtual Reality for Blind People](https://mlsanity.com/paper/2410.14058/) (2024)
- [EnVisionVR: A Scene Interpretation Tool for Visual Accessibility in Virtual Reality](https://mlsanity.com/paper/2502.03564/) (2025)
- [ArtInsight: Enabling AI-Powered Artwork Engagement for Mixed Visual-Ability Families](https://mlsanity.com/paper/2502.19263/) (2025)
- [Accessible Interactive Maps for Visually Impaired Users](https://mlsanity.com/paper/2208.14685/) (2022)
- [TactIcons: Designing 3D Printed Map Icons for People who are Blind or have Low Vision](https://mlsanity.com/paper/2407.20674/) (2024)
- [Making Urban Art Accessible: Current Art Access Techniques, Design Considerations, and the Role of AI](https://mlsanity.com/paper/2410.20571/) (2024)
- [3D Printed Maps and Icons for Inclusion: Testing in the Wild by People who are Blind or have Low Vision](https://mlsanity.com/paper/2501.09204/) (2025)

## Built on by (most cited first, 5 of 5)

- [WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation](https://mlsanity.com/paper/2603.10703/) (2026)
- [StreetDesignAI: Broadening Designer Perspectives Through Multi-Persona Evaluation of Cycling Infrastructure](https://mlsanity.com/paper/2601.15671/) (2026)
- ["Does the cafe entrance look accessible? Where is the door?" Towards Geospatial AI Agents for Visual Inquiries](https://mlsanity.com/paper/2508.15752/) (2025)
- [Making Videos Accessible for Blind and Low Vision Users Using a Multimodal Agent Video Player](https://mlsanity.com/paper/2602.04104/) (2026)
- [URBAN-SPIN: A street-level bikeability index to inform design implementations in historical city centres](https://mlsanity.com/paper/2602.10124/) (2026)

## Abstract (arXiv)

Interactive streetscape mapping tools such as Google Street View (GSV) and Meta Mapillary enable users to virtually navigate and experience real-world environments via immersive 360deg imagery but remain fundamentally inaccessible to blind users. We introduce StreetReaderAI, the first-ever accessible street view tool, which combines context-aware, multimodal AI, accessible navigation controls, and conversational speech. With StreetReaderAI, blind users can virtually examine destinations, engage in open-world exploration, or virtually tour any of the over 220 billion images and 100+ countries where GSV is deployed. We iteratively designed StreetReaderAI with a mixed-visual ability team and performed an evaluation with eleven blind users. Our findings demonstrate the value of an accessible street view in supporting POI investigations and remote route planning. We close by enumerating key guidelines for future work.