mlsanityData to 14 Oct 2026

IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark

arXiv 2507.14449 ↗First posted 19 Jul 2025cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Real-world infrared imagery presents unique challenges for vision-language models due to the scarcity of aligned text data and domain-specific characteristics. Although existing methods have advanced the field, their reliance on synthetic infrared images generated through style transfer from visible images, which limits their ability to capture the unique characteristics of the infrared modality. To address this, we propose IRGPT, the first multi-modal large language model for real-world infrared images, built upon a large-scale InfraRed-Text Dataset (IR-TD) comprising over 260K authentic image-text pairs. The proposed IR-TD dataset contains real infrared images paired with meticulously handcrafted texts, where the initial drafts originated from two complementary processes: (1) LLM-generated descriptions of visible images, and (2) rule-based descriptions of annotations. Furthermore, we introduce a bi-cross-modal curriculum transfer learning strategy that systematically transfers knowledge from visible to infrared domains by considering the difficulty scores of both infrared-visible and infrared-text. Evaluated on a benchmark of 9 tasks (e.g., recognition, grounding), IRGPT achieves state-of-the-art performance even compared with larger-scale models.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 56 references that are in the map. First posted on arXiv 19 Jul 2025.

Builds on

39 of 56 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
27 more
  1. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  2. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  5. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  6. 2020What is being transferred in transfer learning?cited by 368
  7. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  8. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  9. 2024What matters when building vision-language models?cited by 326
  10. 2021Curriculum Learning: A Surveycited by 259
  11. 2023Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancementcited by 231
  12. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  13. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  14. 2021Cross-Domain Gradient Discrepancy Minimization for Unsupervised Domain Adaptationcited by 67
  15. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 65
  16. 2024Task-Customized Mixture of Adapters for General Image Fusioncited by 59
  17. 2023C-SFDA: A Curriculum Learning Aided Self-Training Framework for Efficient Source Free Domain Adaptationcited by 50
  18. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  19. 2024IRSAM: Advancing Segment Anything Model for Infrared Small Target Detectioncited by 46
  20. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 43
  21. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  22. 2023Edge-guided Multi-domain RGB-to-TIR image Translation for Training Vision Tasks with Challenging Labelscited by 30
  23. 2023BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistencycited by 27
  24. 2020LSOTB-TIR:A Large-Scale High-Diversity Thermal Infrared Object Tracking Benchmarkcited by 25
  25. 2024Alignment-Free RGBT Salient Object Detection: Semantics-guided Asymmetric Correlation Network and A Unified Benchmarkcited by 19
  26. 2019Learning a Multi-Domain Curriculum for Neural Machine Translationcited by 16
  27. 2024Alignment-Free RGB-T Salient Object Detection: A Large-scale Dataset and Progressive Correlation Networkcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Promptingcited by 6
  2. 2026Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Taskscited by 4
  3. 2025CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identificationcited by 1
  4. 2026Knowledge-Guided Adversarial Training for Infrared Object Detection via Thermal Radiation Modelingcited by 1
  5. 2026DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoningcited by 1
  6. 2026FusionRS: A Large-Scale RGB-Infrared-Style Remote Sensing Dataset for Cross-Modal Vision-Language Learningcited by 1
  7. 2026ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagerycited by 0
  8. 2026Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Modelscited by 0
  9. 2026Lightweight Multimodal Adaptation of Vision Language Models for Species Recognition and Habitat Context Interpretation in Drone Thermal Imagerycited by 0
  10. 2026Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferabilitycited by 0
  11. 2026MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptationcited by 0
  12. 2026IRSTD-Agent: Agentic Infrared Small Target Detection via Zoom-Guided Interaction Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DiffV2IR: Visible-to-Infrared Diffusion Model via Vision-Language Understanding
  2. 2026FreqKD: Frequency-Decoupled Cross-Modal Knowledge Distillation for Infrared Object Detection
  3. 2025CM-Diff: A Single Generative Network for Bidirectional Cross-Modality Translation Diffusion Model Between Infrared and Visible Images
  4. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain
  5. 2024SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model
  6. 2025Leveraging Language Prior for Infrared Small Target Detection
  7. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignment
  8. 2026SC-Diff: Semantically Calibrated Diffusion for Visible-to-Infrared Image Translation