mlsanityData to 7 Oct 2026

TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint

arXiv 2608.13167 ↗First posted 13 Aug 2026cs.CV cs.AI cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

When visual evidence is occluded or chaotic, models should abstain. In this paper, we show that Vision-Language Models (VLMs) can internally distinguish when abstention is required, but fail to express it anyway. We introduce TRAPSBench, a procedurally generated video benchmark of 1,404 matched physics pairs in which a single targeted change renders the outcome undeterminable from the visual evidence. Furthermore, we introduce Penalized Epistemic Calibration Score (PECS), a new robust metric that requires models to both answer correctly when the outcome is knowable, and abstain when the outcome is not. Across 16 VLMs spanning five families, spontaneous restraint is poor: the best PECS is 0.292. The bottleneck is expression, not perception: linear probes decode answerability from hidden states at up to 0.91 AUROC across physics domains; steering a single-layer void direction causally induces or suppresses abstention. Our results replicate across three open-weight families (Qwen, Gemma, LLaVA). The failure is also more pronounced in visual than textual uncertainty: models detect textual impossibility about 4x more readily than missing visual evidence. Closing this representation--output gap likely requires output-stage interventions.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 42 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

37 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding27 Jan

Other references · 36

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
24 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  4. 2023Steering Language Models With Activation Engineeringcited by 824
  5. 2017Selective Classification for Deep Neural Networkscited by 799
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  10. 2020Rescaling Egocentric Visioncited by 529
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  12. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  13. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  14. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  15. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  16. 2019CoPhy: Counterfactual Learning of Physical Dynamicscited by 80
  17. 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 49
  18. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 44
  19. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  20. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  21. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  22. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Modelscited by 12
  23. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questionscited by 7
  24. 2025VisionTrap: Unanswerable Questions On Visual Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidencecited by 0
  2. 2026New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Modelscited by 0
  3. 2026STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Physics Knowledge in Frontier Models: A Diagnostic Study of Failure Modes
  2. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  3. 2026Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
  4. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility
  5. 2026World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models
  6. 2026Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
  7. 2026CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
  8. 2026CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models