CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
The proliferation of Vision-Language Models (VLMs) in the past several years calls for rigorous and comprehensive evaluation methods and benchmarks. This work analyzes existing VLM evaluation techniques, including automated metrics, AI-based assessments, and human evaluations across diverse tasks. We first introduce Robin - a novel suite of VLMs that we built by combining Large Language Models (LLMs) and Vision Encoders (VEs) at multiple scales, and use Robin to identify shortcomings of current evaluation approaches across scales. Next, to overcome the identified limitations, we introduce CHIRP - a new long form response benchmark we developed for more robust and complete VLM evaluation. We provide open access to the Robin training code, model suite, and CHIRP benchmark to promote reproducibility and advance VLM research.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 38 references that are in the map. First posted on arXiv 16 Jan 2025.
Builds on
22 of 38 references.
Closest ancestors · 2
Same direction, newest first
- 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation12 Jan
- 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models24 Nov
Other references · 20
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2015VQA: Visual Question Answeringcited by 2,489
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2019Towards VQA Models That Can Readcited by 1,696
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
- 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
8 more
- 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
- 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
- 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
- 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 131
- 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
- 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
- 2025Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
- 2024VHELM: A Holistic Evaluation of Vision Language Models
- 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
- 2024What matters when building vision-language models?
- 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
- 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
- 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias