mlsanityData to 7 Oct 2026

MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

arXiv 2608.28384 ↗First posted 28 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce MAP, the first benchmark to evaluate multimodal AI systems as assistants for users with accessibility requirements when planning visits to places in the real world. In our evaluation, systems are presented with requests to verify or recommend a point of interest meeting an accessibility requirement. MAP contains two novel assessments: Claim verification for accessibility planning assesses if information on places and stated accessibility features is supported and identifies places that satisfy requested accessibility features. Visual evidence retrieval for accessibility planning checks if a multimodal AI system can select visual evidence for the requested place and accessibility feature. Our methodology supports comparison of AI systems in a setting where place information and accessibility information can change over time by evaluating systems and refreshing ground truth data at scheduled times. The benchmark is based on automatic rating and human rating for a proportion of responses.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

24 of 37 references.

References · 24

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2024A Survey on LLM-as-a-Judgecited by 1,384
  4. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  5. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  6. 2026Gemma 4 Technical Reportcited by 187
  7. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49
  8. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  9. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  10. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 32
  11. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  12. 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systemscited by 23
12 more
  1. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  2. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?cited by 10
  3. 2025GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidancecited by 8
  4. 2025MuSciClaims: Multimodal Scientific Claim Verificationcited by 8
  5. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  6. 2025CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editingcited by 5
  7. 2026AgentSelect: Benchmark for Narrative Query-to-Agent Recommendationcited by 3
  8. 2024Can ChatGPT assist visually impaired people with micro-navigation?cited by 2
  9. 2025Accessibility Scout: Personalized Accessibility Scans of Built Environmentscited by 2
  10. 2025SceneScout: Towards AI Agent-driven Access to Street View Imagery for Blind Userscited by 1
  11. 2025OmniAcc: Personalized Accessibility Assistant Using Generative AIcited by 1
  12. 2026RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wildcited by 1

Built on by

Not cited yet.