Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Models
Not on the map. Why?
Abstract (arXiv)
When asked which of two events came first, video large language models can fail in two opposite ways: cave to a false claim, or reject a true one. Prior video sycophancy work measures only the first and mitigates it by teaching the model to trust the user less, a fix known in text and image models to worsen the second. In video, both failures come from two causes the literature treats as one: availability, whether the sparse sampled frames contain the two events, and weighting, whether that evidence is trusted over the user. We separate them with two interventions that keep the claim fixed: a frame-preserving reorder that flips the claim's truth, and a sampling-offset shift that captures or misses both events at a fixed frame budget. When the events are missed, the two twins present identical frames, so each of the nine models we evaluate accepts a true and a false claim at the same rate, making Youden's J=0 by construction. Availability is necessary but not sufficient. Five of the nine read the order, yet four of those five still cave to the false claim, so their deference hits a weighting ceiling. Since trust cannot be calibrated over evidence that was never sampled, we propose a reversal test that cancels the model's order prior by scoring the sampled frames forward and reversed, then answers, resamples, or abstains without reading the claim. The test raises the order accuracy to 0.92-1.00 on the models that read the order and abstains rather than guesses on those that cannot.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
32 of 43 references.
References · 32
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
- 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
20 more
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
- 2020Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlationcited by 425
- 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
- 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
- 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
- 2019Cross-task weakly supervised learning from instructional videoscited by 199
- 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
- 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 90
- 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
- 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 68
- 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
- 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
- 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 27
- 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 21
- 2024Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Frameworkcited by 17
- 2024Have the VLMs Lost Confidence? A Study of Sycophancy in VLMscited by 15
- 2025BASIL: Bayesian Assessment of Sycophancy in LLMscited by 13
- 2025Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectoriescited by 11
- 2025SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decodingcited by 10
- 2025PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMscited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.