mlsanityData to 14 Oct 2026

SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters

arXiv 2604.24346 ↗First posted 27 Apr 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are increasingly deployed as evaluators in tasks requiring nuanced image understanding, yet their reliability in scoring alignment between images and text descriptions remains underexplored. We investigate whether small, open-weight VLMs exhibit sycophantic behavior when evaluating image-text alignment: assigning high scores without grounding their judgments in visual evidence. To quantify this phenomenon, we introduce the Bluffing Coefficient (bc), a metric that measures the mismatch between a model's score and its evidence recall. We evaluate six open-weight VLMs ranging from 450M to 8B parameters on a benchmark of 173,810 AI-generated character portraits paired with detailed textual descriptions. Our analysis reveals a significant inverse correlation between model size and sycophancy rate (r = -0.96, p = 0.002), with smaller models exhibiting substantially higher rates of unjustified high scores. The smallest model tested (LFM2-VL, 450M) produced sycophantic evaluations in 22.3% of cases, compared to 6.0% for the largest (LLaVA-1.6, 7B). These findings have direct implications for the deployment of small, open-weight VLMs as automated evaluators within attribute-rich, synthetic image evaluation tasks, where the gap between assigned scores and cited visual evidence is both measurable and consequential.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 33 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

24 of 33 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,833
  2. 2022Training language models to follow instructions with human feedbackcited by 16,029
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,204
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,667
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,101
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,278
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,999
  8. 2015VQA: Visual Question Answeringcited by 2,507
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,435
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,286
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,150
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,049
12 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,079
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 1,056
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 911
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 825
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 715
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 691
  7. 2018Object Hallucination in Image Captioningcited by 675
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 532
  9. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 136
  10. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 102
  11. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 80
  12. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 45

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Models
  2. 2024Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs
  3. 2024Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework
  4. 2026Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation
  5. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
  6. 2026To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
  7. 2025Fooling the LVLM Judges: Visual Biases in LVLM-Based Evaluation
  8. 2024Sycophancy in Large Language Models: Causes and Mitigations