mlsanityData to 14 Oct 2026

UNIV: Unified Foundation Model for Infrared and Visible Modalities

arXiv 2509.15642 ↗First posted 19 Sep 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Joint RGB-infrared perception is essential for achieving robustness under diverse weather and illumination conditions. Although foundation models excel within single modalities, they suffer from substantial cross-modal degradation, an issue we attribute to a pattern shortcut, i.e., a modal bias that prioritizes superficial sensor patterns over underlying semantics. To address this problem, we introduce UNIV, a Unified foundation model for Infrared and Visible modalities. At the core of UNIV lies Patch Cross-modal Contrastive Learning (PCCL), a self-supervised contrastive learning strategy that constructs a unified cross-modal feature space. PCCL employs a frozen pre-trained model to sample pseudo patch pairs based on semantic similarity, and aligns infrared-visible representations by attracting semantically related pairs while repelling unrelated ones. This process simultaneously enhances cross-modal alignment and inter-class semantic separability, guiding the model to focus on semantic structure rather than falling into pattern shortcuts. To further enable cross-modal learning, we introduce MVIP, the most comprehensive visible-infrared benchmark to date, containing 98,992 precisely aligned image pairs across diverse scenes. Extensive experiments demonstrate UNIV's superior performance on infrared tasks (+1.7 mIoU for semantic segmentation and +0.7 mAP for detection), while maintaining competitive accuracy on RGB tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

42 of 61 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2017Mask R-CNNcited by 7,277
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
30 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  3. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  5. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  6. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  7. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  8. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
  9. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  10. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 231
  11. 2020Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learningcited by 152
  12. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  13. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  14. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  15. 2021LasHeR: A Large-scale High-diversity Benchmark for RGBT Trackingcited by 96
  16. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  17. 2020Self-EMD: Self-Supervised Object Detection without ImageNetcited by 70
  18. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  19. 2019Interlaced Sparse Self-Attention for Semantic Segmentationcited by 53
  20. 2019Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentationcited by 36
  21. 2022UniVIP: A Unified Framework for Self-Supervised Visual Pre-trainingcited by 31
  22. 2021SSTN: Self-Supervised Domain Adaptation Thermal Object Detection for Autonomous Drivingcited by 23
  23. 2022Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representationscited by 19
  24. 2023Tagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrievalcited by 18
  25. 2023ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoderscited by 14
  26. 2023R-MAE: Regions Meet Masked Autoencoderscited by 13
  27. 2024InfMAE: A Foundation Model in the Infrared Modalitycited by 11
  28. 2023PAD: Self-Supervised Pre-Training with Patchwise-Scale Adapter for Infrared Imagescited by 5
  29. 2025UNIP: Rethinking Pre-trained Attention Patterns for Infrared Semantic Segmentationcited by 3
  30. 2025MASTER: Multimodal Segmentation with Text Promptscited by 1

Built on by

Not cited yet.