mlsanityData to 14 Oct 2026

M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision

arXiv 2507.16318 ↗First posted 22 Jul 2025cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

RGB-Thermal (RGBT) multispectral vision is essential for robust perception in complex environments. Most RGBT tasks follow a case-by-case research paradigm, relying on manually customized models to learn task-oriented representations. Nevertheless, this paradigm is inherently constrained by artificial inductive bias, modality bias, and data bottleneck. To address these limitations, we make the initial attempt to build a Generalized RGBT MultiSpectral foundation model (M-SpecGene), which aims to learn modality-invariant representations from large-scale broad data in a self-supervised manner. M-SpecGene provides new insights into multispectral fusion and integrates prior case-by-case studies into a unified paradigm. Considering the unique characteristic of information imbalance in RGBT data, we introduce the Cross-Modality Structural Sparsity (CMSS) metric to quantify the information density across two modalities. Then we develop the GMM-CMSS progressive masking strategy to facilitate a flexible, easy-to-hard, and object-centric pre-training process. Comprehensive experiments validate M-SpecGene's generalizability across eleven datasets for four RGBT downstream tasks. The code will be available at https://github.com/CalayZhou/M-SpecGene.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 62 of its 140 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

62 of 140 references.

References · 62

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  5. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  6. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  7. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  8. 2023LightGlue: Local Feature Matching at Light Speedcited by 590
  9. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  10. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 409
  11. 2021Early Convolutions Help Transformers See Bettercited by 396
  12. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
38 more
  1. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 252
  2. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 231
  3. 2023Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentationcited by 129
  4. 2018RGB-T Object Tracking:Benchmark and Baselinecited by 121
  5. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  6. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  7. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  8. 2021LasHeR: A Large-scale High-diversity Benchmark for RGBT Trackingcited by 96
  9. 2022Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baselinecited by 91
  10. 2021Cross-Modality Fusion Transformer for Multispectral Object Detectioncited by 86
  11. 2019PST900: RGB-Thermal Calibration, Dataset and Segmentation Networkcited by 85
  12. 2020Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blockscited by 85
  13. 2020Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problemscited by 83
  14. 2023ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detectioncited by 76
  15. 2024HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Modelcited by 73
  16. 2020RGBT Salient Object Detection: A Large-scale Dataset and Benchmarkcited by 69
  17. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  18. 2021Multimodal Object Detection via Probabilistic Ensemblingcited by 68
  19. 2020Siamese Network for RGB-D Salient Object Detection and Beyondcited by 67
  20. 2019RGB-T Image Saliency Detection via Collaborative Graph Learningcited by 58
  21. 2023C2Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detectioncited by 58
  22. 2018Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detectioncited by 57
  23. 2023Hard Patches Mining for Masked Image Modelingcited by 57
  24. 2020Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Countingcited by 52
  25. 2017A Unified RGB-T Saliency Detection Benchmark: Dataset, Baselines, Analysis and A Novel Approachcited by 51
  26. 2022Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detectioncited by 50
  27. 2018Multispectral Pedestrian Detection via Simultaneous Detection and Segmentationcited by 49
  28. 2016Video Processing from Electro-optical Sensors for Object Detection and Tracking in Maritime Environment: A Surveycited by 47
  29. 2021FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentationcited by 45
  30. 2021Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsingcited by 45
  31. 2020Multi-interactive Dual-decoder for RGB-thermal Salient Object Detectioncited by 44
  32. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  33. 2020HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Imagescited by 41
  34. 2018PTB-TIR: A Thermal Infrared Pedestrian Tracking Benchmarkcited by 34
  35. 2024Visible-Thermal Tiny Object Detection: A Benchmark Dataset and Baselinescited by 27
  36. 2022Glass Segmentation with RGB-Thermal Image Pairscited by 26
  37. 2022ReDFeat: Recoupling Detection and Description for Multimodal Feature Learningcited by 26
  38. 2020LSOTB-TIR:A Large-Scale High-Diversity Thermal Infrared Object Tracking Benchmarkcited by 25

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026SLGNet: Synergizing Structural Priors and Language-Guided Modulation for Multimodal Object Detectioncited by 4
  2. 2026UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmarkcited by 1
  3. 2026M2I2HA: Multi-modal Object Detection Based on Intra- and Inter-Modal Hypergraph Attentioncited by 0
  4. 2026SpectraDINO: Modality-Conditioned Adaptation of RGB Vision Foundation Models Across Infrared Bandscited by 0
  5. 2026Anatomy of a failure: When, how, and why deep vision fails in scientific domainscited by 0
  6. 2026UniTriGen: Unified Triplet Generation of Aligned Visible-Infrared-Label for Few-Shot RGB-T Semantic Segmentationcited by 0
  7. 2026Dual-Correlation Hypergraph Network for Unaligned RGBT Video Object Detection and A Large-scale Benchmarkcited by 0
  8. 2026Repurposing RGB-based Foundation Model for Depth Estimation on Thermal Images Using Hierarchical Supervisioncited by 0