mlsanityData to 14 Oct 2026

Allegory of the Cave: Measurement-Grounded Vision-Language Learning

arXiv 2605.11727 ↗First posted 12 May 2026cs.AI cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models typically reason over post-ISP RGB images, although RGB rendering can clip, suppress, or quantize sensor evidence before inference. We study whether grounding improves when the visual interface is moved closer to the underlying camera measurement. We formulate measurement-grounded vision-language learning and instantiate it as PRISM-VL, which combines RAW-derived Meas.-XYZ inputs, camera-conditioned grounding, and Exposure-Bracketed Supervision Aggregation for transferring supervision from RGB proxies to measurement-domain observations. Using a quality-controlled 150K instruction-tuning set and a held-out benchmark targeting low-light, HDR, visibility-sensitive, and hallucination-sensitive cases, PRISM-VL-8B reaches 0.6120 BLEU, 0.4571 ROUGE-L, and 82.66% LLM-Judge accuracy, improving over the RGB Qwen3-VL-8B baseline by +0.1074 BLEU, +0.1071 ROUGE-L, and +4.46 percentage points. These results suggest that part of VLM grounding error arises from information lost during RGB rendering, and that preserving measurement-domain evidence can improve multimodal reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 17 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

14 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
1 more
  1. 2024Towards RAW Object Detection in Diverse Conditionscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench
  2. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  3. 2025Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video
  4. 2025RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Models
  5. 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations
  6. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning
  7. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
  8. 2025Vision Language Models: A Survey of 26K Papers