mlsanityData to 14 Oct 2026

Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification

arXiv 2608.07920 ↗First posted 8 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multimodal LLM judge panels can cross-reference peers, but a quoted peer judgment may itself be untrusted. We expose source-blind anchoring as a text-level attack surface in vision-language model (VLM) panels. Quoting independent visual judgments creates large anchoring gaps (19--26 percentage points) under both self and peer framing. A matched-content, label-only control changes the broken rate by only -0.17pp (95% CI [-0.68,0.35]), showing that the self/peer label itself does not explain the effect. Under our tested construction, deliberately generated, concise wrong quotes overturn originally-correct verdicts 1.5--2.7× more often than naturally occurring wrong peer statements, with bootstrap 95% CIs excluding parity across two datasets and seven VLM judges. Because the two statement populations differ in selection and form, this ratio measures differential damage under the tested attack rather than a provenance-only causal effect. We then introduce panel-consensus verification, which cross-checks a quote against independently collected blind votes. It blocks 84.9% of fabricated attacks, cuts their net harm by 97.5%, and preserves the positive but statistically inconclusive point estimate for genuine peer information under leave-one-out re-verification. These results identify a low-cost attack surface and a concrete defense for safer multimodal collaborative evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 9 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

7 of 9 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation21 May

Other references · 6

  1. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  2. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  3. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 26
  4. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 4
  5. 2026Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modelingcited by 3
  6. 2026On the Adversarial Robustness of Multimodal LLM Judgescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Social Pressure Breaks Majority Voting in LLM Safety Panels
  2. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
  3. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models
  4. 2026Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks
  5. 2026Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity
  6. 2026Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
  7. 2026Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy
  8. 2025When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injection