UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Vision-Language Models (VLMs) have emerged as powerful tools for image understanding tasks, yet their practical deployment remains hindered by significant architectural heterogeneity across model families. This paper introduces UVLM (Universal Vision-Language Model Loader), a Google Colab-based framework that provides a unified interface for loading, configuring, and benchmarking multiple VLM architectures on custom image analysis tasks. UVLM currently supports two major model families -- LLaVA-NeXT and Qwen2.5-VL -- which differ fundamentally in their vision encoding, tokenization, and decoding strategies. The framework abstracts these differences behind a single inference function, enabling researchers to compare models using identical prompts and evaluation protocols. Key features include a multi-task prompt builder with support for four response types (numeric, category, boolean, text), a consensus validation mechanism based on majority voting across repeated inferences, a flexible token budget (up to 1,500 tokens) enabling users to design custom reasoning strategies through prompt engineering, and a built-in chain-of-thought reference mode for benchmarking. UVLM is designed for reproducibility, accessibility, and extensibility and as such is freely deployable on Google Colab using consumer-grade GPU resources. The paper also presents the first benchmarking of different VLMs on tasks of increasing reasoning complexity using a corpus of 120 street-view images.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 16 references that are in the map. First posted on arXiv 14 Mar 2026.
Builds on
10 of 16 references.
References · 10
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
- 2025Streetscape Analysis with Generative AI (SAGAI): Vision-Language Assessment and Mapping of Urban Scenescited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
- 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
- 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
- 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
- 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
- 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
- 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
- 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies