mlsanityData to 7 Oct 2026

PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models

arXiv 2605.07574 ↗First posted 8 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Mainstream vision-language models (VLMs) fundamentally struggle with severe optical ambiguities, such as reflections and transparent objects, due to the inherent limitations of standard RGB inputs. While polarization imaging captures polarimetric physical parameters that resolve these ambiguities, existing methods are constrained by fixed-format outputs and remain isolated from open-ended reasoning. To bridge this semantic-physical gap, we introduce PolarVLM, the first multimodal framework integrating polarimetric physical parameters into VLMs. By employing a dual-stream architecture and a progressive two-stage training strategy, PolarVLM effectively prevents physical misinterpretations while preserving general visual abilities. Complementing our architecture, we construct PolarVQA, the first benchmark for polarization-aware VQA, featuring 75K physics-grounded instruction-tuning pairs targeting reflective and transparent scenes. Experiments show that PolarVLM surpasses the RGB baseline by 25.4% overall across five evaluation tasks, with remarkable gains of 26.6% in reflection recognition and 34.0% in glass counting, successfully unlocking physics-aware semantic understanding.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 59 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

36 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning20 Apr

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
23 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  5. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  6. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  7. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  8. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  9. 2018Single Image Reflection Separation with Perceptual Lossescited by 98
  10. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  11. 2020Polarized Reflection Removal with Perfect Alignment in the Wildcited by 69
  12. 2021Deep Polarization Imaging for 3D shape and SVBRDF Acquisitioncited by 56
  13. 2021Shape from Polarization for Complex Scenes in the Wildcited by 53
  14. 2022PANDORA: Polarization-Aided Neural Decomposition Of Radiancecited by 52
  15. 2024NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Imagescited by 22
  16. 2025PolarFree: Polarization-based Reflection-free Imagingcited by 18
  17. 2022Transparent Shape from a Single View Polarization Imagecited by 16
  18. 2023Multi-View Azimuth Stereo via Tangent Space Consistencycited by 16
  19. 2024PISR: Polarimetric Neural Implicit Surface Reconstruction for Textureless and Specular Objectscited by 15
  20. 2025EventVL: Understand Event Streams via Multimodal Large Language Modelcited by 13
  21. 2025PIDSR: Complementary Polarized Image Demosaicing and Super-Resolutioncited by 11
  22. 2024Learning to Deblur Polarized Imagescited by 5
  23. 2026LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking
  2. 2026Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models
  3. 2026PolarScale: A Physics-Grounded Benchmark for Radiometrically Consistent RGB-to-Stokes Estimation
  4. 2025RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Models
  5. 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models
  6. 2026Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception
  7. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  8. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs