mlsanityData to 14 Oct 2026

Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles

arXiv 2603.17111 ↗First posted 17 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Ensembling Vision-Language Models (VLMs) from different providers maximizes benchmark accuracy, yet models from the same architectural family share correlated errors that standard voting ignores. We study this structure across 17 VLMs from 8 families on VQAv2, TextVQA, and GQA. Family-correlated errors reduce effective ensemble dimensionality to 2.5-3.6 independent voters and create a Misleading tier (1.5-6.5% of questions) where correlated majority errors destroy accuracy to 0% despite the best model being correct. We propose three family-aware methods. Hierarchical Family Voting (HFV) aggregates within families before voting across them, recovering +18-26 pp on the Misleading tier. QualRCCV, a training-free method weighting models by calibration, family quality, and inverse family size, is the first to beat calibrated voting on all three benchmarks (p<0.05). Learned Candidate Scoring (LCS) trains a cross-validated classifier to re-rank candidate answers using support breadth, family diversity, and model quality, achieving the largest gains: +0.68% VQAv2, +0.61% TextVQA, +2.45% GQA -- all significant -- and is the only learned method that never degrades any benchmark. On VQAv2 test-standard (EvalAI), LCS reaches 87.83% with 12 models, confirming generalization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 23 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

10 of 23 references.

References · 10

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  8. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  9. 2024More Agents Is All You Needcited by 180
  10. 2020In Defense of Grid Features for Visual Question Answeringcited by 155

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models
  2. 2025Test-Time Consistency in Vision Language Models
  3. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  4. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  5. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models
  6. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  7. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models
  8. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models