mlsanityData to 7 Oct 2026

ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery

arXiv 2602.14989 ↗First posted 16 Feb 2026cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision language models (VLMs) achieve strong performance on RGB imagery, but they do not generalize to thermal images. Thermal sensing plays a critical role in settings where visible light fails, including nighttime surveillance, search and rescue, autonomous driving, and medical screening. Unlike RGB imagery, thermal images encode physical temperature rather than color or texture, requiring perceptual and reasoning capabilities that existing RGB-centric benchmarks do not evaluate. We introduce ThermEval-B, a structured benchmark of approximately 55,000 thermal visual question answering pairs designed to assess the foundational primitives required for thermal vision language understanding. ThermEval-B integrates public datasets with our newly collected ThermEval-D, the first dataset to provide dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments. Evaluating 25 open-source and closed-source VLMs, we find that models consistently fail at temperature-grounded reasoning, degrade under colormap transformations, and default to language priors or fixed responses, with only marginal gains from prompting or supervised fine-tuning. These results demonstrate that thermal understanding requires dedicated evaluation beyond RGB-centric assumptions, positioning ThermEval as a benchmark to drive progress in thermal vision language modeling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 64 references that are in the map. First posted on arXiv 16 Feb 2026.

Builds on

29 of 64 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025jina-vlm: Small Multilingual Vision Language Model3 Dec
  2. 2024Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking30 Dec
  3. 2024PaliGemma 2: A Family of Versatile VLMs for Transfer4 Dec
  4. 2024What matters when building vision-language models?3 May
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2024A Survey on LLM-as-a-Judgecited by 1,384
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 313
12 more
  1. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 229
  2. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  3. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  4. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  5. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  6. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilitiescited by 57
  7. 2020Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Countingcited by 52
  8. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  9. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  10. 2020SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streamscited by 21
  11. 2025IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmarkcited by 11
  12. 2024Invisible Gas Detection: An RGB-Thermal Cross Attention Network and A New Benchmarkcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Models
  2. 2026Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception
  3. 2026T-CLIP: Enabling Thermal Perception for Contrastive Language-Image Pretraining
  4. 2026TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation
  5. 2024SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Models
  6. 2026HEARTH: An Object-Centric RGB-Thermal-3D Dataset for Temperature-Aware Robot Manipulation ↗
  7. 2026ThermoAct:Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Making
  8. 2025DarkQA: Benchmarking Vision-Language Models on Visual-Primitive Question Answering in Low-Light Indoor Scenes