mlsanityData to 14 Oct 2026

OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation

arXiv 2509.15096 ↗First posted 18 Sep 2025cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent research on representation learning has proved the merits of multi-modal clues for robust semantic segmentation. Nevertheless, a flexible pretrain-and-finetune pipeline for multiple visual modalities remains unexplored. In this paper, we propose a novel multi-modal learning framework, termed OmniSegmentor. It has two key innovations: 1) Based on ImageNet, we assemble a large-scale dataset for multi-modal pretraining, called ImageNeXt, which contains five popular visual modalities. 2) We provide an efficient pretraining manner to endow the model with the capacity to encode different modality information in the ImageNeXt. For the first time, we introduce a universal multi-modal pretraining framework that consistently amplifies the model's perceptual capabilities across various scenarios, regardless of the arbitrary combination of the involved modalities. Remarkably, our OmniSegmentor achieves new state-of-the-art records on a wide range of multi-modal semantic segmentation datasets, including NYU Depthv2, EventScape, MFNet, DeLiVER, SUNRGBD, and KITTI-360.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 58 of its 83 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

58 of 83 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022A ConvNet for the 2020scited by 3,579
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
38 more
  1. 2020AdaBins: Depth Estimation using Adaptive Binscited by 601
  2. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  3. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  4. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  5. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 377
  6. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  7. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  8. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  9. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  10. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  11. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 158
  12. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  13. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 142
  14. 2022Multimodal Token Fusion for Vision Transformerscited by 136
  15. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  16. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
  17. 2020Deep Multimodal Fusion by Channel Exchangingcited by 118
  18. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  19. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 107
  20. 2022mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentationcited by 95
  21. 2018EV-SegNet: Semantic Segmentation for Event-based Camerascited by 93
  22. 2021N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Camerascited by 92
  23. 2021Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Predictioncited by 90
  24. 2019PST900: RGB-Thermal Calibration, Dataset and Segmentation Networkcited by 85
  25. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 84
  26. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  27. 2021ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentationcited by 77
  28. 2021Is Attention Better Than Matrix Decomposition?cited by 77
  29. 2020RGBT Salient Object Detection: A Large-scale Dataset and Benchmarkcited by 69
  30. 2021DepthTrack : Unveiling the Power of RGBD Trackingcited by 67
  31. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 60
  32. 2019RGB-T Image Saliency Detection via Collaborative Graph Learningcited by 58
  33. 2022CamoFormer: Masked Separable Attention for Camouflaged Object Detectioncited by 58
  34. 2017A Unified RGB-T Saliency Detection Benchmark: Dataset, Baselines, Analysis and A Novel Approachcited by 51
  35. 2020Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentationcited by 48
  36. 2020ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Datacited by 42
  37. 2023MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learningcited by 40
  38. 2022RGB-T Semantic Segmentation with Location, Activation, and Sharpeningcited by 39

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentationcited by 1
  2. 2026RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodingscited by 1
  3. 2026Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observationscited by 0
  4. 2026Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsingcited by 0
  5. 2026Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentationcited by 0