# Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision

arXiv: https://arxiv.org/abs/2508.10972 · HTML: https://mlsanity.com/paper/2508.10972/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 47 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [AXNav: Replaying Accessibility Tests from Natural Language](https://mlsanity.com/paper/2310.02424/) (2023)
- [Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?](https://mlsanity.com/paper/2410.23426/) (2024)
- [VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models](https://mlsanity.com/paper/2402.01735/) (2024)
- [Vision Skills Needed to Answer Visual Questions](https://mlsanity.com/paper/2010.03160/) (2020)
- [Empath-D: VR-based Empathetic App Design for Accessibility](https://mlsanity.com/paper/2503.12933/) (2025)
- [Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies](https://mlsanity.com/paper/2505.22981/) (2025)

## Abstract (arXiv)

Advances in vision language models (VLMs) have enabled the simulation of general human behavior through their reasoning and problem solving capabilities. However, prior research has not investigated such simulation capabilities in the accessibility domain. In this paper, we evaluate the extent to which VLMs can simulate the vision perception of low vision individuals when interpreting images. We first compile a benchmark dataset through a survey study with 40 low vision participants, collecting their brief and detailed vision information and both open-ended and multiple-choice image perception and recognition responses to up to 25 images. Using these responses, we construct prompts for VLMs (GPT-4o) to create simulated agents of each participant, varying the included information on vision information and example image responses. We evaluate the agreement between VLM-generated responses and participants' original answers. Our results indicate that VLMs tend to infer beyond the specified vision ability when given minimal prompts, resulting in low agreement (0.59). The agreement between the agent' and participants' responses remains low when only either the vision information (0.59) or example image responses (0.59) are provided, whereas a combination of both significantly increase the agreement (0.70, p < 0.0001). Notably, a single example combining both open-ended and multiple-choice responses, offers significant performance improvements over either alone (p < 0.0001), while additional examples provided minimal benefits (p > 0.05).