# GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them

arXiv: https://arxiv.org/abs/2609.06058 · HTML: https://mlsanity.com/paper/2609.06058/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 7 of its 8 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 7 of 7)

- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework](https://mlsanity.com/paper/2408.11261/) (2024)
- [A Mechanistic View of Authority Hierarchy in LLM Sycophancy](https://mlsanity.com/paper/2607.00415/) (2026)
- [Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models](https://mlsanity.com/paper/2601.13433/) (2026)
- [Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models](https://mlsanity.com/paper/2509.16149/) (2025)
- [Measuring Agreeableness Bias in Multimodal Models](https://mlsanity.com/paper/2408.09111/) (2024)
- [Investigating VLM Hallucination from a Cognitive Psychology Perspective: A First Step Toward Interpretation with Intriguing Observations](https://mlsanity.com/paper/2507.03123/) (2025)

## Built on by (most cited first, 1 of 1)

- [Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models](https://mlsanity.com/paper/2610.17989/) (2026)

## Abstract (arXiv)

As vision-language models (VLMs) become increasingly capable and are deployed in consequential real-world settings, they must evaluate evidence independently rather than defer uncritically to human authority. We introduce GradeTrap, a controlled evaluation that places two social cues in direct conflict: a student answer, which should attract sycophantic agreement, and a conflicting answer attributed to a peer, teacher, or official answer key, which should attract authority-based deference. Models produce free-form answers while being explicitly instructed to solve independently and ignore all student answers, feedback, and grading marks. We test the models on 60 synthetic real-world trade-off scenarios. Five neutral trials establish a stable model-relative preference, followed by three repetitions of six experimental cues including controls. On the 45-item common intersection across Gemini 3.5 Flash-Lite, GPT-5.6 Luna, and Claude Haiku 4.5, a generic second-answer control yields 5.4% conflicting-answer selection. Relative to that control, pooled within-item changes show no reliable peer-review effect, a 6.9-point teacher-review effect, and a 19.5-point official-key effect. In contrast, a displayed conflicting student answer alone compared to a displayed student reference answer alone only raises selection from 2.2% to 5.2%. Official-key provenance therefore redirects judgements more than a student answer or the generic second-answer control, despite an explicit ignore instruction and an opposing student answer given along with the official key. Effects vary in magnitude across the three models.