# Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impaired

arXiv: https://arxiv.org/abs/2508.03651 · HTML: https://mlsanity.com/paper/2508.03651/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 94 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [Creating General User Models from Computer Use](https://mlsanity.com/paper/2505.10831/) (2025)
- [Proactive Conversational Agents with Inner Thoughts](https://mlsanity.com/paper/2501.00383/) (2024)
- [A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective](https://mlsanity.com/paper/2304.08795/) (2023)
- [Trust in AI and Its Role in the Acceptance of AI Technologies](https://mlsanity.com/paper/2203.12687/) (2022)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments](https://mlsanity.com/paper/2404.02508/) (2024)
- [Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics](https://mlsanity.com/paper/2205.10646/) (2022)
- [Misfitting With AI: How Blind People Verify and Contest AI Errors](https://mlsanity.com/paper/2408.06546/) (2024)
- [Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation System](https://mlsanity.com/paper/2410.04005/) (2024)
- [Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models](https://mlsanity.com/paper/2502.16098/) (2025)
- [Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People](https://mlsanity.com/paper/2503.05899/) (2025)
- [Hand-Priming in Object Localization for Assistive Egocentric Vision](https://mlsanity.com/paper/2002.12557/) (2020)
- [WanderGuide: Indoor Map-less Robotic Guide for Exploration by Blind People](https://mlsanity.com/paper/2502.08906/) (2025)
- [Conversations for Vision: Remote Sighted Assistants Helping People with Visual Impairments](https://mlsanity.com/paper/1812.00148/) (2018)
- [Blind Users Accessing Their Training Images in Teachable Object Recognizers](https://mlsanity.com/paper/2208.07968/) (2022)
- [EditScribe: Non-Visual Image Editing with Natural Language Verification Loops](https://mlsanity.com/paper/2408.06632/) (2024)
- [How Can Generative AI Enhance the Well-being of Blind?](https://mlsanity.com/paper/2402.07919/) (2024)
- [Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM](https://mlsanity.com/paper/2404.02706/) (2024)
- [Memory Reviver: Supporting Photo-Collection Reminiscence for People with Visual Impairment via a Proactive Chatbot](https://mlsanity.com/paper/2501.15408/) (2025)

## Built on by (most cited first, 2 of 2)

- [Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users](https://mlsanity.com/paper/2602.16930/) (2026)
- [More Than Just Access: Generative AI as Communication Intermediary for Blind and Low-Vision Users](https://mlsanity.com/paper/2609.15696/) (2026)

## Abstract (arXiv)

Recent advancements in large multimodal models have provided blind or visually impaired (BVI) individuals with new capabilities to interpret and engage with the real world through interactive systems that utilize live video feeds. However, the potential benefits and challenges of such capabilities to support diverse real-world assistive tasks remain unclear. In this paper, we present findings from an exploratory study with eight BVI participants. Participants used ChatGPT's Advanced Voice with Video, a state-of-the-art live video AI released in late 2024, in various real-world scenarios, from locating objects to recognizing visual landmarks, across unfamiliar indoor and outdoor environments. Our findings indicate that current live video AI effectively provides guidance and answers for static visual scenes but falls short in delivering essential live descriptions required in dynamic situations. Despite inaccuracies in spatial and distance information, participants leveraged the provided visual information to supplement their mobility strategies. Although the system was perceived as human-like due to high-quality voice interactions, assumptions about users' visual abilities, hallucinations, generic responses, and a tendency towards sycophancy led to confusion, distrust, and potential risks for BVI users. Based on the results, we discuss implications for assistive video AI agents, including incorporating additional sensing capabilities for real-world use, determining appropriate intervention timing beyond turn-taking interactions, and addressing ecological and safety concerns.