# Lost in Instructions: Study of Blind Users' Experiences with DIY Manuals and AI-Rewritten Instructions for Assembly, Operation, and Troubleshooting of Tangible Products

arXiv: https://arxiv.org/abs/2602.18630 · HTML: https://mlsanity.com/paper/2602.18630/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 134 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models](https://mlsanity.com/paper/2405.09589/) (2024)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People](https://mlsanity.com/paper/2403.15604/) (2024)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [GenAssist: Making Image Generation Accessible](https://mlsanity.com/paper/2307.07589/) (2023)
- [Misfitting With AI: How Blind People Verify and Contest AI Errors](https://mlsanity.com/paper/2408.06546/) (2024)
- [Making Short-Form Videos Accessible with Hierarchical Video Summaries](https://mlsanity.com/paper/2402.10382/) (2024)
- [Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models](https://mlsanity.com/paper/2502.16098/) (2025)
- [VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interaction](https://mlsanity.com/paper/2310.09611/) (2023)
- [Vid2Coach: Transforming How-To Videos into Task Assistants](https://mlsanity.com/paper/2506.00717/) (2025)
- ["It's Kind of Context Dependent": Understanding Blind and Low Vision People's Video Accessibility Preferences Across Viewing Scenarios](https://mlsanity.com/paper/2403.10792/) (2024)
- [Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People](https://mlsanity.com/paper/2503.05899/) (2025)
- [A Contextual Inquiry of People with Vision Impairments in Cooking](https://mlsanity.com/paper/2402.15108/) (2024)
- [Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models](https://mlsanity.com/paper/2407.19537/) (2024)
- [VideoA11y: Method and Dataset for Accessible Video Description](https://mlsanity.com/paper/2502.20480/) (2025)
- [Improving the Accessibility of Scientific Documents: Current State, User Needs, and a System Solution to Enhance Scientific PDF Accessibility for Blind and Low Vision Users](https://mlsanity.com/paper/2105.00076/) (2021)
- [Non-Visual Cooking: Exploring Practices and Challenges of Meal Preparation by People with Visual Impairments](https://mlsanity.com/paper/2107.05783/) (2021)
- ["It Feels Like Taking a Gamble": Exploring Perceptions, Practices, and Challenges of Using Makeup and Cosmetics for People with Visual Impairments](https://mlsanity.com/paper/2203.08864/) (2022)
- [A Recipe for Success? Exploring Strategies for Improving Non-Visual Access to Cooking Instructions](https://mlsanity.com/paper/2407.19065/) (2024)
- [CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation](https://mlsanity.com/paper/2506.09343/) (2025)
- [MAIDR: Making Statistical Visualizations Accessible with Multimodal Data Representation](https://mlsanity.com/paper/2403.00717/) (2024)
- [Understanding How Blind Users Handle Object Recognition Errors: Strategies and Challenges](https://mlsanity.com/paper/2408.03303/) (2024)
- [SeeChart: Enabling Accessible Visualizations Through Interactive Natural Language Interface For People with Visual Impairments](https://mlsanity.com/paper/2302.07742/) (2023)
- [TADA: Making Node-link Diagrams Accessible to Blind and Low-Vision People](https://mlsanity.com/paper/2311.04502/) (2023)
- [DanmuA11y: Making Time-Synced On-Screen Video Comments (Danmu) Accessible to Blind and Low Vision Users via Multi-Viewer Audio Discussions](https://mlsanity.com/paper/2501.15711/) (2025)

## Abstract (arXiv)

AI tools like ChatGPT and Be-My-AI are increasingly being used by blind individuals. Although prior work has explored their use in some Do-It-Yourself (DIY) tasks by blind individuals, little is known about how they use these tools and the available product-manual resources to assemble, operate, and troubleshoot physical or tangible products - tasks requiring spatial reasoning, structural understanding, and precise execution. We address this knowledge gap via an interview study and a usability study with blind participants, investigating how they leverage AI tools and product manuals for DIY tasks with physical products. Findings show that manuals are essential resources, but product-manual instructions are often inadequate for blind users. AI tools presently do not adequately address this insufficiency; in fact, we observed that they often exacerbate this issue with incomplete, incoherent, or misleading guidance. Lastly, we suggest improvements to AI tools for generating tailored instructions for blind users' DIY tasks involving tangible products.