# MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

arXiv: https://arxiv.org/abs/2608.28384 · HTML: https://mlsanity.com/paper/2608.28384/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Gemma 4 Technical Report](https://mlsanity.com/paper/2607.02770/) (2026)
- [The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input](https://mlsanity.com/paper/2501.03200/) (2025)
- [V-IRL: Grounding Virtual Intelligence in Real Life](https://mlsanity.com/paper/2402.03310/) (2024)
- [Judge Anything: MLLM as a Judge Across Any Modality](https://mlsanity.com/paper/2503.17489/) (2025)
- [MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2410.10139/) (2024)
- [NavBench: Probing Multimodal Large Language Models for Embodied Navigation](https://mlsanity.com/paper/2506.01031/) (2025)
- [AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems](https://mlsanity.com/paper/2505.19623/) (2025)
- [Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users](https://mlsanity.com/paper/2503.22610/) (2025)
- [GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance](https://mlsanity.com/paper/2503.12844/) (2025)
- [Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?](https://mlsanity.com/paper/2505.08468/) (2025)
- [MuSciClaims: Multimodal Scientific Claim Verification](https://mlsanity.com/paper/2506.04585/) (2025)
- [CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?](https://mlsanity.com/paper/2512.16755/) (2025)
- [CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editing](https://mlsanity.com/paper/2507.16407/) (2025)
- [AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation](https://mlsanity.com/paper/2603.03761/) (2026)
- [Can ChatGPT assist visually impaired people with micro-navigation?](https://mlsanity.com/paper/2408.08321/) (2024)
- [Accessibility Scout: Personalized Accessibility Scans of Built Environments](https://mlsanity.com/paper/2507.23190/) (2025)
- [RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild](https://mlsanity.com/paper/2605.10357/) (2026)
- [SceneScout: Towards AI Agent-driven Access to Street View Imagery for Blind Users](https://mlsanity.com/paper/2504.09227/) (2025)
- [OmniAcc: Personalized Accessibility Assistant Using Generative AI](https://mlsanity.com/paper/2509.07220/) (2025)

## Abstract (arXiv)

We introduce MAP, the first benchmark to evaluate multimodal AI systems as assistants for users with accessibility requirements when planning visits to places in the real world. In our evaluation, systems are presented with requests to verify or recommend a point of interest meeting an accessibility requirement. MAP contains two novel assessments: Claim verification for accessibility planning assesses if information on places and stated accessibility features is supported and identifies places that satisfy requested accessibility features. Visual evidence retrieval for accessibility planning checks if a multimodal AI system can select visual evidence for the requested place and accessibility feature. Our methodology supports comparison of AI systems in a setting where place information and accessibility information can change over time by evaluating systems and refreshing ground truth data at scheduled times. The benchmark is based on automatic rating and human rating for a proportion of responses.