# Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users

arXiv: https://arxiv.org/abs/2602.16930 · HTML: https://mlsanity.com/paper/2602.16930/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 112 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Builds on (most cited first, 30 of 48)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://mlsanity.com/paper/2406.06608/) (2024)
- [Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization](https://mlsanity.com/paper/2406.00045/) (2024)
- [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023)
- [Personalized Dialogue Generation with Diversified Traits](https://mlsanity.com/paper/1901.09672/) (2019)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [Yo'LLaVA: Your Personalized Language and Vision Assistant](https://mlsanity.com/paper/2406.09400/) (2024)
- [MyVLM: Personalizing VLMs for User-Specific Queries](https://mlsanity.com/paper/2403.14599/) (2024)
- [On the steerability of large language models toward data-driven personas](https://mlsanity.com/paper/2311.04978/) (2023)
- [Is a Question Decomposition Unit All We Need?](https://mlsanity.com/paper/2205.12538/) (2022)
- [PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions](https://mlsanity.com/paper/2308.05184/) (2023)
- [Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints](https://mlsanity.com/paper/2302.09185/) (2023)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People](https://mlsanity.com/paper/2403.15604/) (2024)
- [EgoBlind: Towards Egocentric Visual Assistance for the Blind](https://mlsanity.com/paper/2503.08221/) (2025)
- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [Personalizing a Dialogue System with Transfer Reinforcement Learning](https://mlsanity.com/paper/1610.02891/) (2016)
- [Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!](https://mlsanity.com/paper/2501.10674/) (2025)
- [Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation System](https://mlsanity.com/paper/2410.04005/) (2024)
- [PrompTHis: Visualizing the Process and Influence of Prompt Editing during Text-to-Image Creation](https://mlsanity.com/paper/2403.09615/) (2024)
- [Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People](https://mlsanity.com/paper/2503.05899/) (2025)
- [Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design](https://mlsanity.com/paper/2407.08882/) (2024)

## Built on by (most cited first, 2 of 2)

- [Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks](https://mlsanity.com/paper/2607.29585/) (2026)
- [Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments](https://mlsanity.com/paper/2609.21828/) (2026)

## Abstract (arXiv)

Prompting and steering techniques are well established in general-purpose generative AI, yet assistive visual question answering (VQA) tools for blind users still follow rigid interaction patterns with limited opportunities for customization. User control can be helpful when system responses are misaligned with their goals and contexts, a gap that becomes especially consequential for blind users that may rely on these systems for access. We invite 11 blind users to customize their interactions with a real-world conversational VQA system. Drawing on 418 interactions, reflections, and post-study interviews, we analyze prompting-based techniques participants adopted, including those introduced in the study and those developed independently in real-world settings. VQA interactions were often lengthy: participants averaged 3 turns, sometimes up to 21, with input text typically tenfold shorter than the responses they heard. Built on state-of-the-art LLMs, the system lacked verbosity controls, was limited in estimating distance in space and time, relied on inaccessible image framing, and offered little to no camera guidance. We discuss how customization techniques such as prompt engineering can help participants work around these limitations. Alongside a new publicly available dataset, we offer insights for interaction design at both query and system levels.