mlsanityData to 7 Oct 2026

VLM-SlideEval: Evaluating VLMs on Structured Comprehension and Perturbation Sensitivity in PPT

arXiv 2510.22045 ↗First posted 24 Oct 2025cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are increasingly used to evaluate multimodal content, including presentation slides, yet their slide-specific understanding remains underexplored {despite their growing role as critics in agentic, model-forward pipelines}. We introduce VLM-SlideEval, an evaluation framework that probes VLMs along three axes: (1) element-level extraction from slide images aligned to ground truth; (2) robustness to controlled perturbations in geometry, style, and text; and (3) higher-level comprehension, such as recovering a deck's narrative order from shuffled slides. Using publicly available decks from Zenodo (https://huggingface.co/datasets/Forceless/Zenodo10K/viewer/default/pptx), we standardize ground-truth element metadata from PowerPoint XML and live renderings into a unified, verifiable schema. Empirically, VLMs underperform on pixel-accurate extraction and show non-trivial agreement, fidelity, and consistency under controlled perturbations, while performing better on single-slide content understanding; however, they do not reliably capture narrative structure across slides. These results highlight the limits of current VLMs for slide evaluation and motivate calibrated, critic-in-the-loop evaluators that drive iterative refinement and selection in agentic pipelines.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 27 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

20 of 27 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Vision Language Models are Biased29 May
  2. 2025MMC: Iterative Refinement of VLM Reasoning via MCTS-based Multimodal Critique15 Apr
  3. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Models29 Oct
  4. 2024Trust but Verify: Programmatic VLM Evaluation in the Wild17 Oct
  5. 2024VHELM: A Holistic Evaluation of Vision Language Models9 Oct
  6. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models3 Jun

Other references · 14

  1. 2020End-to-End Object Detection with Transformerscited by 7,384
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  3. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  5. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  6. 2015End-to-end people detection in crowded scenescited by 140
  7. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  8. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  9. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 68
  10. 2025AutoPresent: Designing Structured Visuals from Scratchcited by 63
  11. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  12. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
2 more
  1. 2025SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generationcited by 5
  2. 2025MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DECKBench: Benchmarking Multi-Agent Frameworks for Academic Slide Generation and Editingcited by 9
  2. 2026EditPPT: Faithful Long-Deck Slide Editing via Structured Tool-Using Multi-Agent with Dual-Modal Validatorscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding
  2. 2026ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correction
  3. 2025Animation Needs Attention: A Holistic Approach to Slides Animation Comprehension with Visual-Language Models
  4. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  5. 2025Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure
  6. 2024Building and better understanding vision-language models: insights and future directions
  7. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?
  8. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone