mlsanityData to 7 Oct 2026

How (not) to ensemble LVLMs for VQA

arXiv 2310.06641 ↗First posted 10 Oct 2023cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper studies ensembling in the era of Large Vision-Language Models (LVLMs). Ensembling is a classical method to combine different models to get increased performance. In the recent work on Encyclopedic-VQA the authors examine a wide variety of models to solve their task: from vanilla LVLMs, to models including the caption as extra context, to models augmented with Lens-based retrieval of Wikipedia pages. Intuitively these models are highly complementary, which should make them ideal for ensembling. Indeed, an oracle experiment shows potential gains from 48.8% accuracy (the best single model) all the way up to 67% (best possible ensemble). So it is a trivial exercise to create an ensemble with substantial real gains. Or is it?

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 33 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

22 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  7. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  11. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  12. 2022Prompting GPT-3 To Be Reliablecited by 258
9 more
  1. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  2. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  3. 2022Language Models are General-Purpose Interfacescited by 96
  4. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  5. 2021A Survey on Ensemble Learning under the Era of Deep Learningcited by 47
  6. 2023Large Language Models Vote: Prompting for Rare Disease Identificationcited by 9
  7. 2023Confidence-based Ensembles of End-to-End Speech Recognition Modelscited by 3
  8. 2023Evaluation of Confidence-based Ensembling in Deep Learning Image Classificationcited by 1
  9. 2023DiversiGATE: A Comprehensive Framework for Reliable Large Language Modelscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024mR2AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQAcited by 19
  2. 2025Enhancing LLM Robustness to Perturbed Instructions: An Empirical Studycited by 17
  3. 2023Distilling Large Language Models for Matching Patients to Clinical Trialscited by 16
  4. 2024A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking LLM Ensembling from the Perspective of Mixture Models
  2. 2024Improving Generalization in Visual Reasoning via Self-Ensemble
  3. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  5. 2023Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models
  6. 2024Does VLM Classification Benefit from LLM Description Semantics?
  7. 2024MouSi: Poly-Visual-Expert Vision-Language Models
  8. 2025VLSM-Ensemble: Ensembling CLIP-based Vision-Language Models for Enhanced Medical Image Segmentation