mlsanityData to 14 Oct 2026

MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation

arXiv 2503.06700 ↗First posted 9 Mar 2025cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Research has focused on Multi-Modal Semantic Segmentation (MMSS), where pixel-wise predictions are derived from multiple visual modalities captured by diverse sensors. Recently, the large vision model, Segment Anything Model 2 (SAM2), has shown strong zero-shot segmentation performance on both images and videos. When extending SAM2 to MMSS, two issues arise: 1. How can SAM2 be adapted to multi-modal data? 2. How can SAM2 better understand semantics? Inspired by cross-frame correlation in videos, we propose to treat multi-modal data as a sequence of frames representing the same scene. Our key idea is to ''memorize'' the modality-agnostic information and 'memorize' the semantics related to the targeted scene. To achieve this, we apply SAM2's memory mechanisms across multi-modal data to capture modality-agnostic features. Meanwhile, to memorize the semantic knowledge, we propose a training-only Semantic Prototype Memory Module (SPMM) to store category-level prototypes across training for facilitating SAM2's transition from instance to semantic segmentation. A prototypical adaptation loss is imposed between global and local prototypes iteratively to align and refine SAM2's semantic understanding. Extensive experimental results demonstrate that our proposed MemorySAM outperforms SoTA methods by large margins on both synthetic and real-world benchmarks (65.38% on DELIVER, 52.88% on MCubeS). Source code will be made publicly available.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 54 of its 80 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

54 of 80 references.

References · 54

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Segment Anythingcited by 8,234
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  5. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  7. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  8. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  9. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 490
  10. 2020BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentationcited by 294
  11. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  12. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
38 more
  1. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  2. 2022Rethinking Semantic Segmentation: A Prototype Viewcited by 140
  3. 2019Fast-SCNN: Fast Semantic Segmentation Networkcited by 125
  4. 2018Robust Classification with Convolutional Prototype Learningcited by 123
  5. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
  6. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
  7. 2024SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memorycited by 94
  8. 2023Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarkscited by 90
  9. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 84
  10. 2024SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Treecited by 80
  11. 2021ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentationcited by 77
  12. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 60
  13. 2020Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentationcited by 48
  14. 2021FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentationcited by 45
  15. 2023Frame-Event Alignment and Fusion Network for High Frame Rate Trackingcited by 43
  16. 2020ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Datacited by 42
  17. 2021ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentationcited by 40
  18. 2023Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objectscited by 40
  19. 2019RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Drivingcited by 39
  20. 2022RGB-T Semantic Segmentation with Location, Activation, and Sharpeningcited by 39
  21. 2022X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentationcited by 38
  22. 2022PP-LiteSeg: A Superior Real-Time Semantic Segmentation Modelcited by 33
  23. 2024ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and Morecited by 33
  24. 2022RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformercited by 31
  25. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 31
  26. 2024360SFUDA++: Towards Source-free UDA for Panoramic Segmentation by Learning Reliable Category Prototypescited by 28
  27. 2024PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Imagescited by 24
  28. 2024Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentationcited by 22
  29. 2023Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Cameracited by 21
  30. 2023Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learningcited by 18
  31. 2023Window Attention is Bugged: How not to Interpolate Position Embeddingscited by 18
  32. 2022Improving Nighttime Driving-Scene Segmentation via Dual Image-adaptive Learnable Filterscited by 17
  33. 2024A Unified Framework with Multimodal Fine-tuning for Remote Sensing Semantic Segmentationcited by 15
  34. 2024Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Expertscited by 15
  35. 2024Segment Anything with Multiple Modalitiescited by 12
  36. 2024MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selectioncited by 12
  37. 2025Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidancecited by 9
  38. 2023Bimodal SegNet: Instance Segmentation Fusing Events and RGB Frames for Robotic Graspingcited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 3.

10 papers.

  1. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  2. 2025Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustnesscited by 18
  3. 2025Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularizationcited by 9
  4. 2026PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learningcited by 7
  5. 2025Split Matching for Inductive Zero-shot Semantic Segmentationcited by 2
  6. 2025EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentationcited by 1
  7. 2025BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentationcited by 0
  8. 2025Med-K2N: Flexible K-to-N Modality Translation for Medical Image Synthesiscited by 0
  9. 20264th Workshop on Maritime Computer Vision (MaCVi): Challenge Overviewcited by 0
  10. 2026Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentationcited by 0