mlsanityData to 14 Oct 2026

SABRE: Scalable and Automated Benchmarking of VLMs under Stress

arXiv 2608.07435 ↗First posted 7 Aug 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 28 references that are in the map. First posted on arXiv 7 Aug 2026.

Builds on

16 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Vision Language Models are Biased29 May
  2. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation6 Jan
  3. 2024Probing Visual Language Priors in VLMs31 Dec
  4. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping11 Oct

Other references · 12

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  7. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  8. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  9. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 45
  10. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
  11. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Imagescited by 5
  12. 2025InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexitycited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
  2. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  3. 2025BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models
  4. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
  5. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  6. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  7. 2025CIVET: Systematic Evaluation of Understanding in VLMs
  8. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation