mlsanityData to 14 Oct 2026

Can Argus Judge Them All? Comparing VLMs Across Domains

arXiv 2507.01042 ↗First posted 23 Jun 2025cs.IR cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) are increasingly used in industry VLM applications such as retrieval systems, content generation platforms, and decision-support workflows, where model selection is commonly guided by benchmark rankings. These rankings are largely determined by retrieval, captioning, and reasoning downstream tasks; however, models with similar task performance often show substantially different behavior across datasets. This creates a Capability-Reliability Gap between benchmark performance and observed model stability. We present ARGUS-EVAL, a capability-reliability-oriented evaluation framework for VLMs that characterizes model behavior through Benchmark Capability P(M), Cross-Dataset Consistency CDC(M), Robustness Retention RR(M), and Efficiency E(M). We evaluate CLIP, BLIP, LXMERT, Gemma-3-4B, and Qwen-2.5VL-3B-Instruct across retrieval, captioning, and reasoning downstream tasks. The results reveal notable differences between capability-oriented and reliability-oriented rankings. Qwen-2.5VL-3BInstruct achieves the strongest overall capability (R@1 = 82.7%, BLEU-4 = 47.2%, CIDEr = 141.6, CDC = 0.91), whereas CLIP records the lowest latency (31 ms) and memory footprint (0.9 GB).

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 44 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

26 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
13 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  6. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  8. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  9. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  11. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 152
  12. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  13. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Are Large Language Models Economically Viable for Industry Deployment?cited by 0
  2. 2026MineTRACE: An Evidence-Grounded Interactive Reasoning System for Mineral Prospectivitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
  2. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  3. 2026Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric
  4. 2026What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs
  5. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  6. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?
  8. 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens