mlsanityData to 14 Oct 2026

Automatic benchmarking of large multimodal models via iterative experiment programming

arXiv 2406.12321 ↗First posted 18 Jun 2024cs.AI cs.CL cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Assessing the capabilities of large multimodal models (LMMs) often requires the creation of ad-hoc evaluations. Currently, building new benchmarks requires tremendous amounts of manual work for each specific analysis. This makes the evaluation process tedious and costly. In this paper, we present APEx, Automatic Programming of Experiments, the first framework for automatic benchmarking of LMMs. Given a research question expressed in natural language, APEx leverages a large language model (LLM) and a library of pre-specified tools to generate a set of experiments for the model at hand, and progressively compile a scientific report. The report drives the testing procedure: based on the current status of the investigation, APEx chooses which experiments to perform and whether the results are sufficient to draw conclusions. Finally, the LLM refines the report, presenting the results to the user in natural language. Thanks to its modularity, our framework is flexible and extensible as new tools become available. Empirically, APEx reproduces the findings of existing studies while allowing for arbitrary analyses and hypothesis testing.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 46 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

44 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models5 Nov
  2. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
29 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  4. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  5. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  6. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  8. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  9. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  10. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  11. 2022TALM: Tool Augmented Language Modelscited by 197
  12. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  13. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  14. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  15. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  16. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  17. 2024A Multimodal Automated Interpretability Agentcited by 63
  18. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 51
  19. 2023Modular Visual Question Answering via Code Generationcited by 48
  20. 2022ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotationscited by 44
  21. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  22. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 31
  23. 2022Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shiftcited by 24
  24. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Modelscited by 23
  25. 2024Can We Talk Models Into Seeing the World Differently?cited by 23
  26. 2023Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoningcited by 22
  27. 2023AmadeusGPT: a natural language interface for interactive animal behavioral analysiscited by 12
  28. 2024Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcementcited by 12
  29. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 42
  2. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Modelscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  3. 2026MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
  4. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
  5. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  7. 2024Benchmarking Large Multimodal Models against Common Corruptions
  8. 2024MileBench: Benchmarking MLLMs in Long Context